Healthcare AI evaluation imagery representing clinician review of medical AI outputs for hallucinations.

Healthcare · · 8 min read

How to test medical AI for hallucinations

Acumen Editorial · Updated November 22, 2026

Every medical AI team reports a benchmark score. Fewer can say whether that score would drop if the model invented a contraindication. New research this autumn shows it often would not. For heads of evaluation, clinical AI leads and post-training teams, that changes what a credible hallucination test looks like.

Why don't standard scores catch medical AI hallucinations?

Because most scores reward what an answer covers, not whether every claim in it is true. A rubric asks whether the response mentions the right drug, the right red flags and the right next step. A response can tick those boxes and still state a wrong dose.

Researchers at the University of Oxford tested this directly in a September 2026 arXiv preprint, "When Rubrics Fail." They took answers to medical questions, injected a single clinically meaningful error into each and asked rubric-based graders to score the original and the altered answer.

The results are uncomfortable:

  • Clinicians judged 80.3% of the injected errors as changing diagnosis or management.
  • Rubrics often gave the original and the error-injected answer the same score. The modal outcome was a tie for every grader tested.
  • HealthBench rubrics discriminated best, at an AUROC of 0.599. HealthBench Professional was near chance.
  • The rubric favored the correct answer in a majority of pairs for only 3 of 13 error types.

The authors' conclusion is plain: "rubric scores alone are insufficient to establish clinical reliability." That does not make rubrics useless. It means a rubric score is a coverage measure, and hallucination needs its own test.

Why does this matter outside the lab?

Because people act on fluent errors. In a JMIR study published November 13, 2026, emergency medicine and trauma surgery physicians at four German hospitals read four GPT-5-generated research abstracts. Two contained false claims. Physicians said they would act on 58% of the false abstracts, and they rated true and false abstracts no differently on accuracy, precision, structure or grammar.

The sample was small, 20 physicians and 80 ratings, and it measured stated intent rather than bedside behavior. The authors call it hypothesis-generating. The direction is still clear: a polished hallucination does not look like a hallucination to a busy reader.

The same pattern shows up in deployed products. An August 2026 preprint audited three commercial ambient AI scribes across 565 notes from 142 consultations. At least one verified failure appeared in 31.3% of notes. Failures clustered in allergy and medication details, invented patient identity, and history written up as a physical exam, including in telephone visits where no exam took place.

Why does the test instrument change the answer?

The scribe audit found something every buyer should know. Changing only the review instruction moved the share of candidate errors that were verified from 9.3% to 79.0%. Depending on the standard applied, between 28% and 97% of sampled notes carried a verified failure.

In other words, a hallucination rate describes the instrument as much as the model. The authors argue buyers should ask not only for an error rate but for the instrument that produced it. Two vendors quoting 5% and 15% may be measuring different things.

That is why a credible test fixes the instrument first: the error taxonomy, the severity scale, who grades and how disagreements are resolved. Then it keeps that instrument stable across model versions.

What methods detect hallucinations, and what do they miss?

No single method is enough. A scoping review in Medical Education Online (November 6, 2026) reached the same view across 21 sources and called for multilayered evaluation. The Oxford paper shows the tradeoffs.

MethodWhat it catchesWhat it missesUse it for
Generic rubricMissing content, structure, safety languageWrong facts inside otherwise complete answersCoverage, not truth
Question-specific rubricMore targeted errors (AUROC 0.61 vs 0.54 generic)Errors the rubric author did not anticipateCore scoring on high-value questions
Taxonomy-based criteriaEvidence fabrication (90.6%), over-triage (86.4%), treatment errors (61.8%)Novel error types; produces false positivesKnown high-risk error classes
Retrieval-based fact check87.5% of injected errors where the claim was checkableFlagged 55.3% of correct answers tooTriage for human review
Seeded-error test setWhether your grader can tell right from wrongNothing on its own; it validates the other methodsCalibrating every automated grader
Clinician reviewClinically meaningful errors, including unexpected onesScale, unless sampled wellGround truth and high-stakes sign-off

The pattern: automated methods are good at narrowing where to look. Clinicians decide what is actually wrong.

How do you build a medical AI hallucination test?

Start from intended use, then build a set that can prove your grader works.

  1. Define the claims that matter. Doses, contraindications, allergies, lab thresholds, guideline recommendations, citations. These are where a wrong fact changes care.
  2. Write an error taxonomy with severity. Fabricated evidence, wrong dose, invented history, false negative, over-triage, under-triage. Have clinicians agree on what counts as management-changing.
  3. Seed errors deliberately. Create paired answers that differ by one clinically meaningful error. If your grader scores them the same, it cannot see hallucinations.
  4. Use question-specific criteria. Tie each criterion to the fact that an error would change, not to general qualities like "accurate."
  5. Add retrieval checks for checkable claims. Route flagged claims to clinicians instead of auto-failing them.
  6. Have specialists grade and adjudicate. Double-read high-severity items. Track agreement and resolve disagreements with a senior reviewer.
  7. Measure reader effect. Test whether clinicians catch seeded errors in fluent output. A low error rate is not safe if readers stop looking.
  8. Freeze the instrument and rerun it. Same taxonomy, prompts and graders on every model, prompt or retrieval change.

Checklist: what to ask a vendor about hallucination testing

  • What error taxonomy do you use, and who agreed on severity?
  • Do you test with seeded errors to prove the grader separates correct from incorrect answers?
  • What is your automated grader's agreement with clinicians on adjudicated items?
  • Are graders licensed clinicians in the relevant specialty?
  • How do you resolve grader disagreement?
  • Can we see the review instructions? Would a different instruction change your error rate?
  • Do you report errors by type and severity, or only one overall score?
  • Can we rerun the same instrument on the next model version?

A vendor that quotes an error rate but cannot answer the sixth question has given you a number without a ruler.

How Acumen helps

We build hallucination test sets for medical AI with verified clinicians. Specialists write question-specific criteria, define error taxonomies and severity, create seeded-error pairs and grade outputs against the evidence. High-severity items are double-read and adjudicated. We calibrate your automated graders against clinician grades and deliver a frozen instrument you can rerun on every release.

See our medical model evaluation service, or read our guide to LLM-as-a-judge for medical AI.

Request a sample dataset Schedule a conversation

FAQ

What is a hallucination in medical AI?

A hallucination is a claim the model states as fact that is false or unsupported, such as an invented contraindication, a wrong dose, a fabricated citation or patient history that never happened. In medical settings the risk is higher because hallucinations usually arrive in fluent, confident language, and readers struggle to tell them apart from correct output.

Do rubric-based benchmarks catch hallucinations?

Often not. A September 2026 University of Oxford preprint injected single clinical errors into answers and found rubrics frequently scored the correct and error-injected versions the same. HealthBench rubrics discriminated best at an AUROC of 0.599, and HealthBench Professional was near chance. Rubrics measure coverage well, so pair them with a dedicated hallucination test.

How do you test a medical LLM for hallucinations?

Define the claims that matter for your use case, write an error taxonomy with severity, and build paired answers that differ by one seeded error. Use question-specific criteria and retrieval checks to flag likely errors, then have specialist clinicians grade and adjudicate. Freeze the instrument and rerun it on every model, prompt or retrieval change.

Why do AI scribe error rates vary so much between studies?

Because the measuring instrument varies. An August 2026 audit of three commercial AI scribes found that changing only the review instruction moved the share of verified candidate errors from 9.3% to 79.0%. Depending on the standard, 28% to 97% of sampled notes carried a verified failure. Always ask which instrument produced a quoted error rate.

Can automated fact-checking replace clinician review?

No. In the Oxford study, a retrieval-based fact check flagged 87.5% of injected errors where the claim was checkable, but it also flagged 55.3% of correct answers. That makes it useful for routing claims to review, not for final judgment. A November 2026 scoping review likewise concluded that no single detection method is sufficient.

Do clinicians notice AI hallucinations?

Not reliably. In a JMIR study published November 2026, emergency and trauma physicians said they would act on 58% of AI-generated abstracts containing false claims, and they rated true and false abstracts similarly on quality. The sample was small, but it shows why evaluations should test whether readers catch seeded errors, not just whether models make them.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights