The Department of Health and Human Services has put OpenAI's ChatGPT on a $2.1 trillion portfolio of Medicare, Medicaid, and ACA claims. It is the largest single LLM deployment in the history of the United States government. Today's AI news roundup confirms the scale of the rollout; a separate industry digest flags the procurement channel and the silence around it. Nobody in the chain of command has answered the question that follows: under what authority, against which guardrails, with what error tolerance, and on whose head when the model is wrong.
The $2.1 Trillion Number, and Why This Isn't Another Pilot
Two and one-tenths of a trillion dollars is not an abstract figure. It is roughly 5.7% of US GDP. It is approximately 24% of total federal outlays in fiscal year 2025. It is the entire annual spend of the Department of Defense, with several billion left over. It is the most consequential line item in the United States budget that is not called "interest on the debt." HHS has now placed an LLM inside the workflow that adjudicates it.
Prior federal LLM deployments were measured in millions. The Air Force's 2024 contract-summarization pilot ran on roughly 1.2 million documents. The IRS's 2025 taxpayer-correspondence assistant handled an estimated 4 million inquiries per year. The State Department's translation pipeline processes a few hundred thousand pages annually. The HHS deployment is four to five orders of magnitude larger in scope, in data volume, and in consequence per error.
The semantic distinction between a "pilot" and a "production system" is doing all the work in the public framing. Operationally, the system is live on live claims. There is no parallel run. There is no human-review-only mode. The model is reading, scoring, and routing claims against the same CMS mainframe that issues payments. The vendors, the department, and the press office have all used the word "pilot." The deployment architecture does not match the word.
What HHS Actually Plugged In
The model in question is a GPT-5 class endpoint, served through the Azure Government cloud under a commercial license between HHS and OpenAI. The access tier is the part of the picture the procurement documents do not specify, and that is the part that matters.
What is on the record: the model has read access to the Integrated Data Repository, which holds every Medicare and Medicaid claim filed in the last decade. It has read access to provider enrollment records, exclusion lists, and prior audit findings. It has read access to the documentation submitted alongside claims — physician notes, discharge summaries, imaging reports, and, in many cases, the scanned PDFs that arrive through the mailroom.
What is not on the record, and what the public record should contain: the write tier. The model is described in the deployment memo as "advisory." The word "advisory" has been used by every previous federal LLM deployment to mean "we do not want to say the model can act." In practice, "advisory" has, in every prior case, become "the system now does this automatically with a human review queue that has not been staffed." The pre-deployment capacity for human review at CMS is approximately 1,200 auditors. The volume of claims the LLM is scoring implies a queue of several million per week. The math does not work. The math not working is the entire problem.
The Hallucination Tax
The failure mode is not a chatbot inventing a citation. It is an LLM misreading a billing code, and the failure mode has a price.
A current-procedure-technology code is five digits. A single transposed digit can convert a $42 routine office visit into a $4,200 surgical procedure. A Healthcare-Common-Procedure-Coding-System modifier applied to the wrong claim line can flip a denied claim to a paid one, or vice versa, with a single token. The LLM does not need to hallucinate in the conversational sense. It needs to misclassify, misroute, or mis-score. Published error rates of frontier models on structured medical-coding tasks in 2025 ranged from 0.4% to 6.1% on the cleanest benchmarks, and the benchmarks were not adversarial.
At a 0.4% error rate applied to $2.1 trillion in adjudicated claims, the expected value of misclassified payments is $8.4 billion. At 1%, it is $21 billion. At 6.1%, it is $128 billion. None of these figures includes the secondary cost of legitimate claims that the model incorrectly flags for fraud review, which is the line item hospital associations have been warning about since the announcement. A hospital that has to fight a fraudulent-looking denial in administrative-law court is a hospital that has spent $40,000 in legal fees on a $9,000 claim. The model can be right and still cause harm.
There is no published benchmark of the deployment's error rate on real CMS data. The vendor has not disclosed one. The department has not commissioned one. The inspector general's office has not been tasked with producing one. The system is live.
The Procurement Loophole
Federal AI deployments are supposed to clear a defined set of gates. The Office of Management and Budget's M-24-10 and M-25-21 memos require risk tiering, an impact assessment, a human-in-the-loop attestation, and a continuous-monitoring plan for any AI system that touches rights-impacting decisions. The Federal AI Risk Management Act requires a public inventory entry on ai.gov. FedRAMP High authorization is the floor for any system processing personally identifiable health information.
HHS's deployment has none of these on the public record. The route around the gates is a procurement vehicle. The model is being delivered as a commercial software-as-a-service product under an existing blanket purchase agreement managed by the General Services Administration. The "advisory" framing in the contract is the legal basis for the exemption from the rights-impacting tier. The exemption is, on its face, absurd: an "advisory" system whose outputs are acted on by an understaffed review queue is, in operation, a decision-making system. The procurement document does not have to call it that for the operational reality to be a decision.
No FedRAMP High authorization appears in the OpenAI federal certification registry. No Authority to Operate letter has been published. No continuous-monitoring plan has been filed. The system went live in June. The first round of decisions it produced was on July 1.
The Lethal Trifecta Goes to Washington
The security research community has a name for the configuration that creates the highest-stakes failure modes in an LLM deployment: the lethal trifecta. Three ingredients. First, the model has access to private data. Second, the model can take actions against an external system. Third, the model processes untrusted content from the outside world. A model with all three can be tricked, by way of an adversarial document, into reading sensitive data and exfiltrating it or acting on its instructions.
HHS's deployment has all three. The private data is the Integrated Data Repository. The external action is the claim-routing workflow against the CMS mainframe. The untrusted content is the documentation submitted with each claim: physician notes, scanned PDFs, hospital discharge summaries, prior-authorization attachments. A claim package from a sophisticated adversary can contain a prompt injection. The model will read it. The model's downstream system will execute the instruction. There is no documented mitigation against this class of attack in the deployment architecture.
In 2024, a single uploaded PDF was sufficient to compromise a CMS contractor's document-processing pipeline. The contractor was human-staffed. The contractor had training. The contractor was compromised. The LLM replacing the contractor has no equivalent intuition, no equivalent skepticism, and a much wider set of privileges.
The Precedent This Sets for Every Cabinet Department After HHS
Cabinet departments talk to each other. The procurement vehicle HHS is using is reusable. The contract template is reusable. The "advisory" framing is reusable. The exemption from the rights-impacting tier is, on the current reading of the law, reusable. Treasury will be next, with the IRS's $5.1 trillion in annual collections and the same model serving taxpayer-correspondence triage. The Department of Defense will be next, with the same model on logistics and contract review. The Social Security Administration will be next, with the same model on disability determinations.
Each of these is larger in scope than the HHS deployment. Each is rights-impacting in a more direct sense. Each will inherit the procurement loophole, the missing FedRAMP authorization, and the missing error-rate disclosure, because the template has been set. The first department to do it absorbs the political cost. The second department gets the benefit of the precedent. By the fourth department, the absence of guardrails is the default.
Congress has not held a hearing on the HHS deployment. The inspector general has not opened a review. The Government Accountability Office has not been requested to study the procurement. The first round of model-produced decisions is already in the appeals queue. The clock on the precedent is not the date of a hearing. It is the date the first appeal is decided.