Healthcare · · 8 min read
How to evaluate AI for prior authorization
Acumen Editorial · Updated November 22, 2026
Prior authorization is where AI meets coverage decisions, and the scrutiny has arrived at once. Medicare's AI pilot is under formal review, states are writing clinician-review rules into law, and a new benchmark shows the best agents finish only about a third of real admin tasks. For health-system clinical AI leads, evaluation teams and the vendors who sell to them, the question is no longer whether AI can help with prior authorization. It is how to prove it does so safely.
Why is AI prior authorization under scrutiny now?
Three things moved in 2026.
- Medicare's WISeR pilot. CMS launched WISeR on January 1, 2026 in six states, covering about 6.4 million traditional Medicare beneficiaries and 13 categories of services. Records released through an Electronic Frontier Foundation lawsuit, reported by Tech Times on November 3, 2026, show CMS sent formal compliance actions to four of the six AI vendors within the first weeks and months. The issues cited included slow determinations, blank denial letters and data mismatches.
- Federal timing rules. Under the CMS Interoperability and Prior Authorization Final Rule (CMS-0057-F), impacted payers must decide urgent requests within 72 hours and standard requests within seven calendar days from 2026, give a specific reason for each denial, and post prior authorization metrics publicly. The Prior Authorization API is due by January 1, 2027.
- State clinician-review laws. Becker's Payer Issues counted seven state AI health insurance laws passed in 2026. Washington's SB 5395 says only a licensed physician or health professional may deny on medical necessity grounds. Colorado's HB 1139, effective January 1, 2027, requires decisions based on the individual's clinical history and periodic audits of the AI.
KFF adds that Medicare Advantage rules already bar medical necessity decisions made by software that ignores individual circumstances, and that patients have filed class actions over algorithmic denials. The message for buyers is plain. Speed, accuracy and the reasoning behind a decision will all be examined, and an evaluation needs to cover all three.
What do benchmarks say about AI prior authorization agents?
The clearest public evidence is HealthAdminBench, described by Stanford HAI on October 19, 2026. It contains 135 tasks: 60 prior authorizations, 60 appeals and denials and 15 durable medical equipment orders, checked by 1,698 verifiers. The best model completed 36.3% of tasks end to end. One model passed 82.8% of subtasks but completed only 26.7% of whole tasks.
That gap is the main lesson. Agents handle each step well enough and then lose information between steps, documents and systems. Stanford HAI identified that cross-step carryover as the dominant failure. In prior authorization, a missed lab value in step two becomes a wrong criteria match in step five, and a step-level score will not show it.
Two more points matter. Benchmark tasks are not your payer policies, your specialty mix or your documentation quality, so a public score is a starting signal, not a validation. And benchmarks grade task completion, not whether a recommended denial would survive a physician peer-to-peer or an appeal.
What should an AI prior authorization evaluation measure?
Measure the whole decision, from packet intake to the letter a clinician signs. We use six dimensions.
| Dimension | What to test | Who grades | Failure it catches |
|---|---|---|---|
| Evidence extraction | Does the system find every relevant fact in the chart, including scanned notes and outside records? | UR nurses | Missed labs, prior therapies or imaging |
| Criteria application | Does it map facts to the right policy criteria for this payer and service? | UR nurses, specialty physicians | Wrong policy version, misread step-therapy rules |
| Individual circumstances | Does the recommendation reflect this patient, not a population default? | Specialty physicians | Template denials that ignore comorbidities |
| Denial-reason quality | Is each adverse reason specific, accurate and traceable to a criterion? | Physician reviewers | Blank, generic or unsupported denial letters |
| Escalation and abstention | Does it route uncertain or adverse cases to a clinician and stop when evidence is missing? | Physician reviewers | Silent auto-denials, overconfident approvals |
| End-to-end completion and time | Does the full case finish correctly within the 72-hour or seven-day clock? | Evaluation lead | Backlogs, dropped handoffs, missed deadlines |
Report results by service category and by outcome. Errors that wrongly deny care and errors that wrongly approve it carry different harms and should never be averaged into one accuracy number.
How do you build a prior authorization test set?
- Pull real request packets. Sample de-identified cases across your highest-volume and highest-risk service categories, including incomplete and messy submissions. Clean packets flatter the model.
- Freeze the policies. Record the exact policy version for each case. Criteria change, and an evaluation run against shifting rules cannot be repeated.
- Write gold decisions with clinicians. Have two UR nurses or specialty physicians independently decide each case, cite the criteria they used and draft the denial reason where relevant. Adjudicate disagreements with a senior physician and keep the record.
- Seed hard cases. Add cases where one buried fact flips the decision, cases that should be pended for missing information, and cases where the policy is silent. These test abstention and carryover.
- Grade full cases and steps. Score each step for diagnosis, but report end-to-end completion as the headline, as HealthAdminBench does.
- Rerun on every change. Model, prompt, retrieval and policy updates all require a rerun on the frozen set before release.
Does the evaluation differ for payers and providers?
The core instrument is the same, but the emphasis shifts. Payer-side systems make or recommend coverage decisions, so wrongful denials, individual-circumstance review and denial-reason quality carry the most weight, and state audit rules apply directly. Provider-side systems assemble and submit requests, so evidence extraction and completeness matter most. A packet that omits a prior therapy will be pended or denied no matter how good the payer's model is.
Both sides share the clock. A request that stalls inside an agent workflow still counts against the 72-hour or seven-day window, so time-to-decision belongs in every test run. Both sides also need the same honesty about abstention. A system that pends a case for missing information is behaving correctly, and the evaluation should reward it.
Checklist: what to ask an AI prior authorization vendor
- Do you report end-to-end case completion, or only step accuracy?
- Who wrote your gold decisions, and what are their credentials and specialties?
- How do you measure agreement between clinician graders, and how are disagreements adjudicated?
- Are wrongful denials and wrongful approvals reported separately?
- Can every adverse recommendation be traced to a specific criterion and chart fact?
- Does the system ever finalize a denial without a licensed clinician's review?
- How do you meet the 72-hour and seven-day decision clocks, and what happens when a case stalls?
- Which policy version did each test case use, and can we rerun it ourselves?
- How will you support state rules that require individual clinical review and periodic audits?
A vendor that cannot answer the first four questions has not evaluated the decision. It has evaluated the steps.
How Acumen helps
Acumen builds clinician-graded evaluation sets for healthcare AI. For prior authorization, that means verified utilization review nurses and specialty physicians who write gold decisions on real-format packets, cite the criteria behind each call, draft reference denial reasons and adjudicate disagreements. We deliver frozen test sets, grading rubrics and agreement statistics your team can rerun on every release. See our medical model evaluation capability, or read how to evaluate AI agents in the EHR.
FAQ
What is AI prior authorization?
AI prior authorization uses software, often language-model agents, to gather chart evidence, match it to payer coverage criteria and recommend or submit a decision on whether a service is approved before it is delivered. It is used by payers reviewing requests and by providers preparing them. Most current rules require a licensed clinician to make or review adverse decisions.
Can AI deny a prior authorization request on its own?
Increasingly, no. Medicare Advantage rules require medical necessity denials to be reviewed by a health care professional, according to KFF. Several states go further. Washington's SB 5395 allows only a licensed physician or health professional to deny on medical necessity grounds, and Iowa and Alabama bar AI from being the sole basis of a denial.
How accurate are AI agents at prior authorization tasks?
Public evidence says not accurate enough to run unsupervised. On HealthAdminBench, described by Stanford HAI in October 2026, the best model completed 36.3% of 135 admin tasks end to end. One model passed 82.8% of subtasks but finished only 26.7% of whole tasks, because information was lost between steps and systems.
What federal deadlines apply to prior authorization decisions?
Under CMS-0057-F, impacted payers must decide urgent requests within 72 hours and standard requests within seven calendar days from 2026, give a specific reason for denials and publish prior authorization metrics each year. The Prior Authorization API is due by January 1, 2027. The denial-reason requirement does not apply to drug prior authorizations.
Who should grade an AI prior authorization evaluation?
Use the clinicians who make these decisions today. Utilization review nurses are best for evidence extraction and criteria matching. Specialty physicians should grade individual-circumstance judgments and denial reasons in their own field. Use two independent graders per case, report their agreement, and have a senior physician adjudicate disagreements so the gold set can be defended.
What is the CMS WISeR model?
WISeR is a CMS pilot that uses AI vendors to run prior authorization for selected services in traditional Medicare. It launched January 1, 2026 in Arizona, New Jersey, Ohio, Oklahoma, Texas and Washington, covers 13 service categories and runs through 2031. Records reported in November 2026 show CMS issued compliance actions to four of its six vendors.