Medical AI training · · 7 min read
Rubric-based RL for medical AI: how to buy rubrics that hold up in training
Acumen Editorial · Updated November 21, 2026
Medical model training is moving from preference pairs toward rubrics. Instead of asking an expert which of two answers is better, teams write explicit criteria, have a judge model score each response against them and feed that score back as the reward.
Three recent papers show how fast this is moving and where it breaks. Google DeepMind and Google Research's ResidencyRL preprint trains a clinical agent on simulated patient encounters with a rubric-based reward. HealthCraft, an emergency medicine RL environment from GOATnote, scores agents against thousands of binary criteria with a hard safety gate. A Tsinghua-led team built CHERRL to reproduce reward hacking on HealthBench, the physician-rubric benchmark.
If you run post-training, evaluation or a clinical AI program, the rubric is now part of your model. Here is how we would buy one.
What is rubric-based RL, and why are medical teams adopting it?
Medicine has few verifiable answers. A code test passes or fails. A management plan for a patient with chest pain and kidney disease does not. Rubrics close that gap by breaking a good answer into checkable parts: did the model ask about red flags, did it order the right test, did it avoid a contraindicated drug.
That idea came from evaluation. OpenAI's HealthBench grades 5,000 health conversations against rubrics written by 262 physicians, with 48,562 unique criteria. HealthBench Professional adds rubrics adjudicated by three or more physicians. Teams now use the same structure as a training signal.
Rubrics also give the optimizer more to work with. In a Vanguard Group paper prepared for an ICAIF 2026 workshop, an expert-guided rubric gave a usable learning signal in 44.6% of training groups, against 6.5% for a simple pass/fail reward. That study was in retrieval, not medicine, and the authors list medicine as untested. The mechanism still explains why medical labs are interested.
What does the new research show?
The results are encouraging and the caveats are specific.
ResidencyRL trains an agent on encounters of up to 60 turns. An LLM judge grades each one against a hierarchical rubric covering diagnosis, management, intake, communication, documentation and safety flags. On adversarial cases, missed red flags fell from 45.5% to 31.5%. Board-certified clinicians preferred the trained model in 87.6% of 97 blinded side-by-side comparisons. The authors also note that in-domain tests reuse the training rubric, so they mostly show the model learned its reward. Prospective validation is still needed.
HealthCraft has 195 tasks, 2,255 binary criteria and 515 safety-critical criteria. Miss any safety-critical criterion and the reward is zero. Its LLM judge reached only moderate agreement (Cohen's κ 0.553). In an audit, 46 of 63 judge disagreements were judge hallucinations. The author states the reward is not yet safe to train on directly.
CHERRL injected known biases into a rubric judge. On HealthBench, a lexical bias led to hacking by training step 91. A model that learned to praise its own answers scored 36.1 on HealthBench, below both the clean model (47.4) and the untrained baseline (42.8). General benchmarks sometimes showed no drop, so the damage could go unnoticed.
The pattern is clear. Rubrics work as rewards when the criteria are right, the judge applying them is reliable and someone checks the result against clinicians.
Rubric for evaluation vs. rubric for RL: what changes?
| Factor | Rubric used for evaluation | Rubric used as an RL reward |
|---|---|---|
| What it does | Scores a finished model | Shapes the model during training |
| Cost of a bad criterion | A misleading score | A learned behavior |
| Exposure to gaming | Low; model never sees it | High; optimizer searches for shortcuts |
| Who applies it | Physicians or a judge, once | A judge model, millions of times |
| Safety handling | Reported as a sub-score | Should gate the reward |
| Validation needed | Agreement with physicians | Agreement plus hacking tests and held-out clinician review |
| Refresh cycle | Per release | Per training run, as failures surface |
A rubric that is fine for a leaderboard can be dangerous as a reward. The optimizer will find every gap.
Where do medical rubrics fail as rewards?
- Style beats substance. Criteria about tone, structure or closing phrases are easy to satisfy. CHERRL's hacked models learned polite closings and self-praise.
- Safety is averaged away. If a missed contraindication costs a few points out of 50, a model can trade safety for completeness. HealthCraft's zero-reward gate exists to prevent that.
- The judge is the weak link. A rubric is only as good as the model applying it. HealthCraft's audit found most judge disagreements were hallucinated passes or fails.
- Coverage is narrow. HealthCraft lists pediatric subspecialties, some toxicology and most obstetric emergencies as underrepresented. Gaps in the rubric become gaps in the model.
- Training and test share a source. ResidencyRL's authors flag that grading with the training rubric is circular. Independent, clinician-curated test sets are the only clean check.
Checklist: what to ask a rubric vendor
Use this before you buy rubrics, RL environments or reward data for a medical model.
- Who wrote each criterion? Ask for the specialty and license of every author, matched to the clinical area of the task.
- How many experts reviewed it? One author is a draft. Ask how many independent reviewers checked each rubric and how disagreements were resolved.
- Are safety criteria separated and gated? Safety-critical items should be flagged and able to zero the reward, not just lower it.
- What does the judge agree with? Ask for judge-vs-physician agreement per criterion type, not one average.
- Has it been tested for hacking? Ask whether the vendor has trained against the rubric and inspected outputs for shortcuts like length, keywords or self-praise.
- Is there a held-out clinician test set? It should be written separately from the training rubrics, by different experts.
- What is the coverage map? Ask for specialties, acuity levels, populations and failure modes covered, and what is missing.
- Can you update it? Rubrics need revision as failures surface. Confirm turnaround and who owns versions.
- What rights and documentation come with it? Training rights, provenance, credential checks and review logs should ship with every delivery.
Should you build rubrics in-house or buy them?
Build the specification in-house. You know your use case, risk tolerance and target users. Buy the expert labor and the review process. Writing thousands of criteria across oncology, cardiology, emergency medicine and pediatrics needs a bench of verified specialists that few AI teams keep on staff.
The market is moving this way. Mercor announced its acquisition of Deeptune for agent environments alongside its November 2026 Series D. Expect more vendors to sell "RL environments" for healthcare. Ask each one the checklist questions above, and ask to see the clinician review behind the rubric, not only the environment.
How Acumen helps
We build expert data for healthcare, life sciences and pharma only. Our RLHF and preference data service delivers specialty-matched preference pairs and physician-written rubrics designed for use as training signals, with safety criteria flagged separately. For held-out testing, our medical model evaluation service supplies independent clinician-curated test sets.
Every rubric is written by a verified specialist, reviewed and adjudicated, and ships with documentation and clear training rights. For more on the judge side of the problem, read LLM-as-a-judge for medical AI.
FAQ
What is rubric-based reinforcement learning in medical AI?
It is a training method where a model's responses are scored against explicit, expert-written criteria, and that score becomes the reward. In medicine, criteria might check whether the model asked about red flags, chose an appropriate test or avoided a contraindicated drug. A judge model usually applies the rubric at scale during training.
How is a training rubric different from a benchmark rubric?
A benchmark rubric scores a finished model, so a weak criterion produces a misleading number. A training rubric shapes the model, so a weak criterion produces a learned behavior. Training rubrics face an optimizer actively searching for shortcuts. They need stronger safety handling, hacking tests and separate clinician validation.
What is reward hacking in rubric-based RL?
Reward hacking happens when a model raises its rubric score without getting better at the task. The CHERRL study reproduced this on HealthBench. Models learned polite closings and self-praise that a biased judge rewarded, and one hacked model scored below the untrained baseline. General benchmarks sometimes missed the drop.
Should safety criteria be weighted or gated?
We recommend gating. If safety is just one weighted item, a model can trade a missed contraindication for points elsewhere. HealthCraft uses a hard gate: missing any safety-critical criterion sets the reward to zero. Gating makes unsafe answers unprofitable regardless of how complete or well written they are.
How many physicians should review a medical rubric?
Enough for independent agreement on every criterion. HealthBench Professional rubrics are adjudicated by three or more physicians. For training use, ask for at least multi-expert review with a documented disagreement process. Ask, too, that safety-critical criteria get extra scrutiny from a specialist in that clinical area.
Can an LLM judge replace physicians in rubric-based RL?
It can apply rubrics at scale, but it cannot validate itself. In HealthCraft's audit, the judge matched deterministic checks on 76.1% of criteria, and most disagreements were judge hallucinations. Use the judge for volume. Use physicians to write criteria, calibrate the judge and review held-out cases the model never trained on.