Clinical evaluation imagery representing physician grading and medical AI assessment.

Medical AI evaluation · · 8 min read

LLM-as-a-judge for medical AI: when to trust it and when you need physicians

Acumen Editorial · Updated November 20, 2026

Every medical AI team hits the same wall. Physician grading is slow and expensive, and a model ships weekly. So teams hand the grading to another model. That choice now shapes most published health benchmarks and most internal release gates.

Four recent studies give buyers a clearer picture of where that works. A Harvard Medical School and Beth Israel Deaconess-led preprint introduced PrecepTron, a judge trained on physician scores. A University of Lübeck-led study measured nine LLM raters against ten physicians. A May 2026 scoping review covered 134 healthcare LLM-as-a-judge studies. A Johns Hopkins team tested judges on completeness. Together they say the same thing: judges scale physician judgment. They do not replace it.

If you lead evaluation, post-training or clinical AI safety, here is how we would decide what to automate.

What is LLM-as-a-judge in medical AI?

An LLM judge is a model that scores another model's output. You give it the case, the response and a rubric, and it returns a grade. In medicine the rubric usually covers accuracy, completeness, safety and communication.

The appeal is obvious. The PrecepTron authors note that "blinded physician evaluation has been considered by many to be the gold standard." It is also the bottleneck. Their dataset alone took 9,217 physician scores from 11 grading physicians across seven studies.

A judge turns that one-time effort into something you can rerun on every checkpoint. The question is whether its scores still mean what a physician's would.

How well do LLM judges agree with physicians?

Better than skeptics expect on narrow tasks. Worse than vendors claim on open-ended ones.

StudyTaskPhysician baselineBest judge resultWeak spot
Scoping review, 134 studies (May 2026)Mixed healthcare tasksVariesMedian Cohen's κ 0.78; median agreement 0.83Raw agreement overstates reliability
Lübeck-led MedQADE studyGerman exam-style answersPhysician κ 0.612Gemini 3 Flash κ 0.694 vs. 0.709 physician referenceSelf-preference; almost no abstention
PrecepTron preprintLandmark diagnostic cases67% physician-physicianFine-tuned judge 61%; frontier judges 28–68%Off-the-shelf judges disagree with each other
Johns Hopkins completeness studyPatient-facing chatbot answersClinician annotationsAUC 0.49–0.66Near chance on missing information

Three patterns stand out.

Scoped tasks align. The scoping review finds that well-scoped tasks with explicit rubrics align better than open-ended or subjective ones. On short exam answers, the Lübeck team found a strong judge almost matched physicians.

Off-the-shelf judges drift apart. On the PrecepTron landmark cases, frontier judges given only the published rubric agreed with physicians on 30% (GPT-5) and 28% (Claude) of responses. Gemini reached 68%. Their mean scores differed by up to 17 points. Switch the judge and you change the leaderboard.

Completeness is hard. The Johns Hopkins team found judges and clinicians often reach the same verdict for different reasons. At the threshold needed to catch 90% of incomplete answers, clinicians would still review most of the dataset. The judge saved no work.

Where do LLM judges fail?

The failures are specific, and each one has a fix.

  • Self-preference. In the Lübeck study, four of five models rated their own answers more favorably than out-of-family judges did, by 7.91 to 16.29 percentage points. Never let a model family grade itself.
  • No sense of uncertainty. Physician abstention rose with difficulty, from a median of 2.68% to 13.74%. Seven of nine LLM evaluators abstained in at most 0.51% of cases. A judge that is never unsure is wrong with confidence on the hardest items.
  • Rubric gaming. The PrecepTron authors found models often hit high rubric scores from the first few sentences of a case. That may reflect pattern matching rather than reasoning.
  • Misleading metrics. The scoping review cites a study with 0.80 raw agreement but κ of 0.59. When one grade dominates, agreement looks better than it is.
  • Fluency over substance. The review lists rewarding fluent answers, evaluator hallucination and prompt sensitivity among common failure modes.

The Lübeck authors are blunt about scope: "These results do not establish clinical safety."

Does calibrating the judge on physician scores help?

Yes, and it is the most useful finding for buyers. The PrecepTron team found prompt-only methods, including few-shot prompting, did not consistently improve their open-source judge. Fine-tuning on a small set of physician scores did.

The fine-tuned 32B judge was trained on only 2 to 46 unique cases per task and runs on a single 80GB GPU. It matched physician agreement at 92% on NEJM clinicopathological conferences. It reproduced the headline findings of five published studies without new human grading.

The catch is in the limitations. The judge inherits the preferences of the panel that trained it. Where physicians agreed only modestly, the judge's ceiling was low too. Calibration data quality sets the limit. Who grades, from which specialty, under which rubric, decides what your judge learns.

How should you set up a physician-calibrated judge?

Use this checklist before a judge gates a release.

  • Scope the task. Write one rubric per task with explicit criteria. Do not reuse a general rubric across diagnosis, triage and patient messaging.
  • Collect a physician calibration set. Specialty-matched physicians grade a stratified sample, with at least two graders per item and adjudication of disagreements.
  • Measure physician agreement first. Report κ or α between physicians. That is the ceiling for your judge.
  • Use a judge from a different model family than the model under test, or an ensemble.
  • Report chance-corrected metrics. κ or α, not raw agreement alone, broken out by difficulty.
  • Add an abstain path. Route low-confidence, high-risk or disputed items to physicians.
  • Hold out a physician-graded test set the judge never trains on.
  • Recalibrate on a schedule. New model versions, new tasks or a new judge model all trigger fresh physician grades.

This matches the scoping review's recommendation to treat judges as evaluation co-pilots. They should route uncertain or high-stakes cases to experts, be validated prospectively and be monitored for drift.

What should you ask an evaluation vendor about its judges?

1. Which judge model grades our outputs, and is it from the same family as our model?

2. What physician scores calibrated it, from which specialties, and how many graders per item?

3. What is physician-physician agreement on this task, and how close is the judge?

4. Which metric do you report, and is it chance-corrected?

5. What share of items goes to physicians, and how is that decided?

6. How often is the judge recalibrated, and on what data?

For the wider evaluation plan, see our guide on how to evaluate a clinical LLM.

How Acumen helps

We supply the physician layer a judge needs. Our medical model evaluation service delivers specialty-matched physician graders, task-specific rubrics, adjudicated calibration sets and held-out test sets. Teams use them to train and audit judges and to grade the cases judges should not.

Every grader is credential-verified and every item is reviewed by physicians with documented agreement. You get a judge you can defend to a clinical safety board, and a refresh path for each release.

Request a sample dataset Schedule a conversation

FAQ

What is LLM-as-a-judge in medical AI?

It is the practice of using one language model to grade another model's medical answers against a rubric, usually for accuracy, completeness, safety and communication. Teams use it to rerun evaluation on every checkpoint without new physician grading each time. It works best when the judge has been calibrated against physician scores for that specific task.

Can LLM judges replace physician graders?

No. A May 2026 scoping review of 134 healthcare studies recommends treating judges as co-pilots that route uncertain or high-stakes cases to experts. Judges align well on scoped tasks with explicit rubrics. They struggle on open-ended reasoning and completeness, and a Johns Hopkins study found them near chance at detecting missing information in chatbot answers.

How accurate are LLM judges compared with physicians?

It depends on the task. The scoping review reports a median Cohen's kappa of 0.78 across ten studies. A Lübeck-led study found one judge close to physician agreement on exam-style answers. On open-ended diagnostic cases, a Harvard-led preprint found off-the-shelf frontier judges agreed with physicians on as few as 28 percent of responses.

What is self-preference bias in LLM judges?

It is a judge's tendency to grade its own model family more favorably. In a Lübeck-led study of nine evaluators, four of five models rated their own answers higher than out-of-family judges did, by about 8 to 16 percentage points. The fix is simple: use a judge from a different model family or an ensemble of families.

How many physician-graded cases do you need to calibrate a judge?

Fewer than most teams expect, if they are well chosen. The PrecepTron preprint fine-tuned its judge on only 2 to 46 unique cases per task and still reproduced five published studies. Grader quality matters more than volume. The judge inherits the preferences of its panel, so specialty match and adjudication decide the result.

Which metric should you use to compare a judge with physicians?

Use a chance-corrected metric such as Cohen's kappa or Krippendorff's alpha, and report physician-physician agreement as the ceiling. Raw agreement can mislead when one grade dominates. The scoping review cites a study showing 0.80 agreement but a kappa of only 0.59. Break results out by case difficulty too.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights