A dark grid of glass evaluation tiles, a row lit green for passed cases and one lit blue, suggesting case-by-case clinical model evaluation.

Model Evaluation · · 8 min read

How to evaluate a clinical LLM

Acumen Editorial · Updated November 20, 2026

Every major model company now sells a healthcare product. OpenAI launched OpenAI for Healthcare in January 2026. Anthropic followed with Claude for Healthcare days later. On November 10, Google put Gemini for healthcare into a US preview for hospitals and insurers, with connectors to Epic and Oracle Health. Tech-ish reported that the launch came with no published benchmarks or independent evaluations.

That leaves the evaluation to you. If you lead clinical AI at a health system, or post-training at a healthcare AI company, the question is no longer whether a model can pass an exam. It is whether it does the work, every time, without the one error that matters.

Here is how we approach it.

Why aren't medical exam scores enough?

Exam scores measure recall on clean, multiple-choice questions. Clinical work is multi-step, messy and full of context. Stanford HAI put it plainly in October: "A licensing exam asks for the right answer." Real care asks for the right action.

Stanford's new agent benchmarks show the gap. PhysicianBench tests agents on 100 multi-step clinical tasks across 21 specialties inside a simulated EHR. The best model, GPT-5.5, completed 46% of tasks on a single attempt. It completed only 28% of tasks in all three runs. On HealthAdminBench, which covers prior authorizations, appeals and equipment orders, GPT-5.4 passed 82.8% of subtasks but finished only 26.7% of whole tasks.

Strong parts do not add up to a reliable whole. A model that aces a vignette can still fail the workflow.

What should a clinical LLM evaluation measure?

Start from the job the model will do, then measure the dimensions that decide whether it is safe to ship. We use six.

DimensionWhat it asksHow to measure it
Task completionDid the model finish the whole job?End-to-end pass rate on realistic, multi-step cases
Clinical reasoningWas the differential, workup and guideline use sound for this patient?Physician-written rubric criteria per case
Action fidelityDid the required action actually happen?Check the state of the system, not the note
ReliabilityDoes it succeed every time?Rerun each case three or more times; report all-runs pass rate
Worst-case safetyHow bad are the failures?Severity-graded review of every failed case
RegressionDid the new version get worse anywhere?Same frozen set, rerun on every release

Action fidelity deserves a note. In PhysicianBench, 62% of GPT-5.5's action failures were cases where the agent wrote that it had done something in its note without placing the order in the EHR. A text-only grader would have scored that as a pass.

How do you build a clinical evaluation set?

The set is the asset. Public benchmarks tell you how models compare. A private set tells you whether this model works for your patients, your specialties and your workflows.

Write cases from real practice. Practicing physicians in the relevant specialty write cases from their own experience: presentation, workup, constraints and a reference answer. Cover the specialties your product touches, from oncology to emergency medicine.

Write rubrics, not answer keys. OpenAI's HealthBench set the pattern in 2025: 5,000 realistic conversations graded against conversation-specific rubrics written by 262 physicians from 60 countries. Rubrics reward what matters, such as ordering the differential sensibly or flagging a red-flag symptom, and penalize what causes harm.

Keep it held out. A test set that leaks into training measures memory, not skill. Store it separately, restrict access and document its independence. FDA's January 2025 draft guidance on AI-enabled device software asks sponsors to describe exactly this: how data was collected and processed, and how test data stays independent.

Include the hard cases on purpose. A 2025 literature survey on HealthBench found that strong average scores can mask dangerous weaknesses in high-stakes situations. Seed the set with rare presentations, ambiguous histories and cases where the right move is to escalate.

Who should grade the outputs?

Physicians should write the criteria. Grading can then be split by risk.

Low-risk, well-specified criteria can be scored by an LLM grader that you calibrate against physician grades. Stanford's MedHELM framework uses an LLM jury scored against clinician-defined criteria. High-risk cases, disagreements and every failure should go to specialists. When two physicians disagree, a third adjudicates. HealthBench Professional uses rubrics adjudicated by three or more physicians.

Track inter-rater agreement. If your graders do not agree with each other, your scores mean less than they look.

What changes when the model is an agent?

Agents read records, call tools and take actions. That adds failure modes a chat evaluation never sees. Stanford found that agents on both PhysicianBench and HealthAdminBench struggled to carry information across steps, documents and systems. The authors pointed to agent architecture and memory more than model size.

So test the full workflow in a realistic environment. Give the agent the same records, portals and tools it will use in production, with synthetic patients. Grade each step and the final state. Check that it stopped and escalated when it should have. And test the harness, not only the model: the same model can pass or fail depending on how it is wired into your systems.

How often should you re-evaluate?

Every time the model changes. Vendors ship new model versions often, and a product built on Gemini, Claude or GPT inherits each change. Freeze your evaluation set, rerun it on every release and compare case by case, not just on the average.

This is also how regulated products work. FDA's final Predetermined Change Control Plan guidance, issued in December 2024, sets out how AI models can be updated after clearance. A standing evaluation set is the evidence that an update did not break anything.

Checklist: what to ask a clinical AI vendor about evaluation

Use this list in your next vendor review or RFP.

  • Which tasks did you evaluate, and do they match our workflows?
  • Were cases written by practicing physicians in our specialties?
  • Is the test set held out from training, and how do you prove it?
  • Do you grade outcomes in the system, or only the text?
  • What is the all-runs pass rate, not just the best single run?
  • How many physicians graded each case, and what was inter-rater agreement?
  • Can we see the worst failures, with severity?
  • Will you rerun our set on every model update and share the diff?
  • Is the evaluation documented well enough for our security, procurement and regulatory reviews?

If a vendor cannot answer most of these, you are buying a demo, not a validated product.

How Acumen helps

We build private clinical evaluation sets and run them for you. Practicing physicians, matched by specialty, write the cases and rubrics. Specialists grade the outputs, a third reviewer adjudicates disagreements, and we document provenance and test-data independence for procurement and regulatory review. Then we rerun the set on every release so you see exactly what changed.

We are healthcare, life sciences and pharma only. That focus is the point: the physicians grading your model know your specialty, and the documentation is built for the reviews your product has to pass.

See how it works on our Medical Model Evaluation page, or read about the clinical AI training data that closes the gaps an evaluation finds. More guides are in Insights.

Request a sample dataset Schedule a conversation

FAQ

How do you evaluate a clinical LLM?

Test it on the real tasks it will perform, using a private set of physician-written cases with rubrics. Grade outcomes rather than prose, rerun each case several times to measure reliability, and review every failure by severity. Freeze the set and rerun it on every model release so you can see regressions case by case, not just as a change in the average score.

Is a high score on a medical licensing exam enough?

No. Exams test recall on clean questions. Clinical work is multi-step and messy. Stanford's PhysicianBench found the best model completed 46% of clinical tasks in a simulated EHR on one attempt and only 28% across all three runs. Strong exam or subtask performance does not guarantee the model will finish the whole job reliably.

What is rubric-based evaluation?

Rubric-based evaluation scores each model response against criteria written for that specific case, rather than against a single correct answer. Physicians write criteria that reward sound reasoning and penalize harmful advice. OpenAI's HealthBench popularized the approach in 2025 with rubrics written by 262 physicians. It captures safety and completeness that multiple-choice accuracy misses.

Should physicians or LLMs grade clinical AI outputs?

Both, split by risk. Physicians write the criteria and grade high-risk cases, disagreements and every failure. An LLM grader can score well-specified, low-risk criteria once it has been calibrated against physician grades. Track inter-rater agreement between graders. When two physicians disagree, a third should adjudicate before the score counts.

How do you keep a test set out of training data?

Store the evaluation set separately from any training pipeline, restrict who can access it and never share it with the model vendor for tuning. Document how it was built and how its independence is maintained. FDA's January 2025 draft guidance on AI-enabled device software asks sponsors to describe test-data independence, so this documentation also supports regulatory review.

How often should a clinical LLM be re-evaluated?

On every model change. Products built on frontier models inherit each new version, and behavior can shift without notice. Rerun the same frozen evaluation set on every release and compare results case by case. FDA's Predetermined Change Control Plan guidance governs how cleared AI models can be updated, and a standing evaluation set is the evidence an update is safe.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights