Highlighted clinical AI evaluation tiles representing tests of medical reasoning, omissions and action failures.

Healthcare · · 8 min read

LLM medical reasoning errors: why general models still fail in the clinic

Acumen Editorial · Updated November 20, 2026

LLM medical reasoning errors are failures in how a model reaches a clinical conclusion, not in what it knows. General models now top medical exams, yet they anchor on first details, stay confident when information is missing, omit critical steps and leave plans unexecuted. Exams rarely catch these errors. Physician-built, sequential evaluations do.

Do general models really fail on medical reasoning?

On knowledge tests, no. A 2026 Nature Medicine study from NYU Langone Health and UT Austin compared GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 with two purpose-built clinical tools, OpenEvidence and UpToDate Expert AI. Gemini scored 97.4% on 500 USMLE-style MedQA questions. On 500 HealthBench items, GPT scored 88 against 62.6 for OpenEvidence, as reported by TechTarget. General models also led on 100 real physician queries graded blind by 12 clinicians.

Two caveats matter. eMarketer notes that frontier models may have seen exam questions in training, and that OpenAI built HealthBench. OpenEvidence called the data "contaminated." Even taken at face value, the result shows something narrower than clinical competence: these models know medicine and write well about it.

Reasoning is a different test. It is what a model does when information arrives in pieces, when the right answer is missing, or when a plan has to become an order. That is where the 2026 evidence shows consistent, specific failures.

What are the most common LLM medical reasoning errors?

We group the published evidence into five error types. Each one survives high exam scores.

Error typeWhat it looks like2026 evidenceHow to test for it
AnchoringThe model fixes on the first detail and ignores later evidenceUNSW-led triage study: best model agreed with ground truth at QWK 0.295 at commit; clinicians reached 0.887–0.929Sequential cases where key findings arrive late
Overconfidence under missing informationThe model picks an answer when it should say it cannot tellMacquarie University study: with the correct option removed, hallucination ran 62–81% across five modelsCases with deliberately missing data and an explicit abstain option
OmissionThe answer is accurate but leaves out a critical stepNOHARM (Stanford/Harvard): omissions made up more than 80% of severe errorsPhysician rubrics that score what must be present, not only what is wrong
Plan-action gapThe model writes the right plan but never executes itStanford HAI PhysicianBench: 62–73% of action failures were described in notes but never orderedAgentic tasks in a sandboxed EHR with checks on actions taken
Inconsistency across runsThe same task passes once and fails the next timePhysicianBench: best model completed 46% of tasks once, 28% across three runsRepeated runs scored on worst case, not average

Anchoring on the chief complaint

A September 2026 preprint from the University of New South Wales and three Sydney hospitals tested six models, including GPT-5.5 and Claude Opus 4.6, on emergency triage. Given a full record, models agreed moderately with ground truth. Given the conversation turn by turn, agreement fell and never recovered as evidence arrived. Removing the chief-complaint turns shifted scores. Removing later turns had almost no effect. Predictions clustered at mid-acuity levels, and the most urgent cases were under-triaged. The authors put it plainly: "Deploying LLMs for ED triage based on offline benchmarks alone misses this sequential failure."

Confidence that does not track accuracy

The Macquarie University study ran 500 MedMCQA questions through GPT-5, GPT-4o, GPT-4o-mini, LLaMA-3 and Mistral. Telling models the information might be incomplete did not meaningfully lower their confidence. When the correct option was replaced with "insufficient information," most models still chose a remaining answer. Unsafe confident errors ranged from 18.6% for GPT-5 to 82.8% for LLaMA-3. The authors call for abstention mechanisms and benchmarks that test incomplete information.

Good plans, no orders

Stanford HAI's October 2026 MedHELM update added PhysicianBench: 100 long-horizon tasks across 21 specialties in a sandboxed EHR. Reasoning gaps, mostly incomplete reasoning and near misses, caused about half of failed checkpoints. In most action failures, the agent described the right step in its notes and never placed the order. Stanford summarized it as "beautifully written words into a good plan, but never actually got the work done." On HealthAdminBench, GPT-5.4 passed 82.8% of subtasks but completed 26.7% of whole tasks.

Why don't exam benchmarks catch these errors?

Exams hand the model a complete, tidy case and a list of options. Real care gives partial information in sequence, rewards saying "I need more data" and ends with an action. A multiple-choice score measures recall under ideal conditions. It cannot show anchoring, abstention or execution.

Rubric-based benchmarks help. OpenAI's HealthBench grades 5,000 multi-turn conversations against rubrics written by 262 physicians. But a 2025 literature survey on HealthBench warned that strong averages can mask dangerous weaknesses in high-stakes situations. In healthcare, the worst case matters more than the mean.

The field is moving in this direction. Google's health AI evaluation framework, published November 4, 2026, scores accuracy, safety and usefulness by combining closed benchmarks, automated graders and expert human review across common and high-risk scenarios. Stanford invites health systems to contribute workflow-level benchmarks to MedHELM. The common thread is physician judgment applied to realistic, messy tasks.

How do you test a model for medical reasoning errors?

Use this checklist before a clinical model reaches production or a procurement decision.

  • Sequential cases. Release findings turn by turn. Score the answer at each checkpoint, not only at the end.
  • Late-breaking evidence. Put the decisive finding late in the case to expose anchoring.
  • Missing-information cases. Remove key data or the correct option. Reward abstention and the right follow-up question.
  • Omission rubrics. Have physicians list what a safe answer must contain, then score for presence.
  • Action checks. For agents, verify the order, referral or note was actually placed.
  • Repeated runs. Run each task at least three times. Report the worst run.
  • Specialty coverage. Weight cases toward the specialties and acuity levels your users see.
  • Credentialed adjudication. Record who graded each item, their specialty and how disagreements were resolved.
  • Held-out data. Keep test cases out of any training or tuning set, and document that independence.

That last point is not optional for regulated products. FDA's January 2025 draft guidance on AI-enabled device software asks sponsors to describe data collection and processing and to show test-data independence.

What should buyers ask before trusting a reasoning claim?

Ask any model or data vendor four questions. Were the test cases sequential or complete records? Was there a correct way to abstain? Who wrote and adjudicated the rubrics, and in which specialties? Is the reported number an average or the worst of several runs? A vendor that cannot answer has measured knowledge, not reasoning.

How Acumen helps

We build the evaluation data that exposes these errors. Credential-verified clinicians write sequential cases from their own practice, design missing-information traps, author omission rubrics and adjudicate model outputs by specialty. Every item carries its reviewer trail and stays out of training sets. See Medical model evaluation, or have clinicians probe for unsafe answers with Healthcare AI red teaming. For reading vendor scores, start with How to read medical AI benchmark claims.

Request a sample dataset Schedule a conversation

FAQ

What are LLM medical reasoning errors?

They are failures in how a model reaches a clinical conclusion rather than gaps in medical knowledge. The main types in 2026 research are anchoring on early details, overconfidence when information is missing, omitting critical steps, writing plans without executing them, and inconsistent results across repeated runs. Each can coexist with very high exam scores.

If frontier models beat clinical AI tools on benchmarks, why worry?

Because those benchmarks mostly test knowledge on complete cases. The NYU Langone study showed general models lead on MedQA and HealthBench, but reviewers flagged possible training contamination. Sequential, agentic and missing-information tests from Stanford HAI, UNSW and Macquarie University show the same model families failing in ways exams do not measure.

What is diagnostic anchoring in an LLM?

Anchoring is fixing on the first information and failing to update as new evidence arrives. In a 2026 UNSW-led triage study, model accuracy depended on the chief-complaint exchange, and later conversation turns barely changed predictions. The most urgent patients were under-triaged. Testing requires sequential cases where the decisive finding comes late.

How do you measure overconfidence in a medical model?

Compare stated confidence with actual accuracy, then remove the information needed to answer. A well-calibrated model should abstain or ask for more data. In a Macquarie University study, uncertainty cues did not lower model confidence, and most models still picked an answer when the correct option had been removed.

Can automated graders replace physicians in evaluation?

Not fully. Automated graders scale scoring, but physicians define what a safe, complete answer contains and adjudicate edge cases. Google's November 2026 health AI evaluation framework pairs automated graders with expert human review, and HealthBench rubrics were written by 262 physicians. Use automation for volume and clinicians for ground truth.

What data do we need to test for LLM medical reasoning errors?

You need physician-written sequential cases, missing-information cases, omission rubrics, agentic tasks with action checks and repeated-run protocols, all held out from training. Each item should record the reviewer's specialty and how disagreements were resolved. That documentation also supports FDA expectations on test-data independence.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights