Expert healthcare data imagery representing clinician-graded medical LLM evaluation.

Healthcare · · 11 min read

Best medical LLM evaluation companies (2026)

Acumen Editorial · Updated November 22, 2026

Acumen publishes this list. We rank ourselves first and describe every other provider from public sources.

Every serious medical AI team now asks the same question: who grades the model? Benchmarks are cheap and fast. Model judges are cheaper still. Neither tells you whether a cardiologist would sign off on the answer.

The research this fall is blunt. A November 2026 paper from Google researchers and clinical co-authors lays out three evaluation methods for health AI: closed-question benchmarks, LLM autoraters and expert human review. It ranks expert review highest for realism and calls for autoraters to be validated against human judgment. The same paper notes that two physicians often diverge on a rating by 30 to 40 percent or more, which is why adjudication matters.

Model judges are not a shortcut yet. In a Harvard and Beth Israel Deaconess preprint, off-the-shelf frontier judges given only a published rubric disagreed with physicians, and their mean scores on one case set differed by up to 17 percentage points. In npj Digital Medicine this year, Williams and colleagues found the best AI jury matched clinician ratings on only 5 of 11 criteria.

So the vendor you pick for clinician grading is part of your evidence. Here is how the field compares.

How we ranked them

We scored each provider on six criteria that decide whether a medical evaluation holds up.

  1. Clinician credential verification. Are license, board status and specialty checked per grader, and is the method public?
  2. Specialty depth. Can the provider staff the exact specialties and care settings in your intended use?
  3. Grading and adjudication rigor. Are rubrics physician-written, are disagreements resolved by a third reviewer, and is agreement reported?
  4. Agent and workflow evaluation. Can they grade multi-step work, not only single answers?
  5. Compliance and documentation. Is PHI handled under stated controls, and does the output document how test data was built and kept independent?
  6. Healthcare focus. Is healthcare the core business or one domain among many?

The best medical LLM evaluation companies

1. Acumen

What they do: Verified expert data for AI in healthcare, life sciences and pharma, including clinician-graded evaluation sets, physician-written rubrics, adjudicated scores and red-team cases.

Best for: Evaluation leads, post-training teams and health-system clinical AI leads who need specialty-matched grading they can defend in review.

Verticals and experts: Healthcare, life sciences and pharma only. Physicians, nurses, pharmacists and other clinicians across major specialties.

Proof: Multiple graders per item, inter-rater agreement reported, fast first sample.

Where we fit less well: We are not a self-serve labeling platform. Teams that want to run high-volume image labeling on their own tooling will pair us with an annotation vendor.

2. Centaur Labs

What they do: Medical and health data annotation and model evaluation, delivered through a large network of scored experts.

Best for: Teams that need medical labels and health-answer evaluation across text, image, audio, video and waveform data.

Verticals and experts: Medical devices, life sciences, insurance, consumer health and LLM developers. Its site cites 100,000+ experts, with specialists in areas such as dermatology, ophthalmology and medical coding.

Sourced fact: Centaur says it scores every annotator continually, then weights and combines their outputs. Its CLEAR-Bench compares models on accuracy, urgency, safety and consumer preference using crowd preferences, clinician reviews and model judges.

Where it fits less well: Its method aggregates many scored annotators. Teams that want a fixed panel of board-certified specialists per item, with a named adjudication record, should ask how panels are formed.

3. Scale AI

What they do: Data, evaluation and post-training services for AI labs and enterprises, with a research arm, Scale Labs, that publishes benchmarks.

Best for: Teams evaluating clinical agents that want benchmark-grade task design at scale.

Verticals and experts: Horizontal, with clinician contributors for healthcare work.

Sourced fact: Scale Labs released CliniCARE-Bench on August 26, 2026: 750 patient cases built from 25 clinician-authored scenarios on real MIMIC-IV records, testing 16 agentic systems. Top four-way accuracy was 76.1%, and reference verdicts matched independent clinician judgment on 91% of defined-label cases.

Where it fits less well: Healthcare is one domain among many. Buyers should ask how specialists are matched and adjudicated on their own tasks.

4. Mercor

What they do: A marketplace that recruits and pays experts to train and evaluate AI models, with a dedicated healthcare and clinical track.

Best for: Frontier labs that need many clinicians quickly across broad programs.

Verticals and experts: Horizontal. Its healthcare page lists physicians across specialties, nurses, pharmacists and certified medical coders, for work it groups as evaluate, create, test and review.

Sourced fact: Mercor's APEX-1 General Practitioner track uses 100 held-out cases graded against expert rubrics. The top model scored 76.2%, per Mercor's September 2026 update.

Where it fits less well: Its healthcare page states credential requirements but does not describe how credentials are verified. Teams often design their own clinical rubrics and QA.

5. Snorkel AI

What they do: AI data development services that supply expert-curated data to label, refine and evaluate domain-specific models.

Best for: Teams that want research-grade evaluation design alongside expert data.

Verticals and experts: Horizontal, with healthcare among its domains. It lists SOC 2 and HIPAA compliance.

Sourced fact: Snorkel recruited the physician annotators for MedPAIR, in which trainees and board-certified physicians labeled relevant sentences in 933 clinical questions. Physicians and LLMs agreed on roughly 50 to 60% of relevant sentences, per Snorkel's October 2026 write-up.

Where it fits less well: Its public pages give few healthcare-specific service details. Buyers should confirm specialty staffing and adjudication for their use case.

6. iMerit

What they do: Data annotation and AI training services, now part of ExlService, with medical AI and life sciences practices.

Best for: Regulated annotation programs that need audit trails and clinical coders.

Verticals and experts: Medical AI, pharma and life sciences, among other sectors. Its site cites board-certified experts and clinical coders.

Sourced fact: iMerit Scholars, launched October 26, 2026, connects developers with more than 30,000 vetted professionals across 118 countries, per SiliconANGLE.

Where it fits less well: Its medical emphasis is annotation, such as imaging and records. Teams that need physicians to write rubrics and grade free-text reasoning should confirm that staffing.

7. Surge AI

What they do: Human data and evaluation for frontier labs, built on a large expert network.

Best for: Frontier lab teams running hard evaluations across many fields at once.

Verticals and experts: Horizontal. Its workforce page lists 200,000+ PhDs across 500+ disciplines and names NIH medical advisors among its experts.

Sourced fact: Surge describes its work as evaluations that go beyond benchmarks, per its workforce page.

Where it fits less well: Its public pages do not describe a medical credential check or healthcare-specific evaluation process.

How do these medical LLM evaluation companies compare?

ProviderCredential verificationSpecialty depthGrading rigorAgent evaluationCompliance and documentationHealthcare focus
AcumenPer-grader license and specialty checksMatched to intended usePhysician rubrics, adjudication, agreement reportedMulti-step workflow gradingPHI-free sets, independence documentedCore
Centaur LabsPerformance-scored expertsNamed medical specialtiesWeighted aggregation of scored annotatorsNot statedSOC 2 Type II, HIPAACore
Scale AINot stated publiclyClinician-authored scenariosClinical Board calibration in CliniCARE-BenchAgentic benchmark publishedNot stated for healthcareOne of many domains
MercorRequirements stated, method not describedBroad clinical rolesExpert rubrics in APEXNot statedNot statedOne of many domains
Snorkel AIBoard-certified annotators in MedPAIRNot statedResearch-grade designAgent testing described generallySOC 2, HIPAAOne of many domains
iMeritBoard-certified experts citedMedical coders, imagingThree-tier vetting (Scholars)Not statedAudit trails citedOne of several sectors
Surge AINot describedNIH medical advisors namedNot described for healthNot statedNot statedOne of many domains

How do you choose a medical LLM evaluation company?

Start with the risk of the use case. A consumer symptom checker, an ambient scribe and an oncology agent need different graders, different rubrics and different thresholds. The Google paper recommends setting passing thresholds by clinical risk rather than one universal cutoff.

Then run every shortlisted vendor through the same questions:

  • How do you verify each grader's license, board status and specialty, and can you show it per item?
  • Who writes the rubric, and is it specific to each case?
  • How many clinicians grade each item, and how are disagreements resolved?
  • Do you report inter-rater agreement, and what was it on your last healthcare project?
  • Can you grade a multi-step agent run, including the actions it took?
  • If you use model judges, how were they calibrated against physicians?
  • How do you keep the test set out of our training data and free of PHI?
  • What documentation do we receive for an FDA submission or a health-system review?

Ask for a sample graded set on your own use case before you sign. A vendor's sample tells you more than its deck.

Most teams end up with two partners: a broad provider for volume and a vertical provider for the grading that has to stand up in review. The mistake is asking a volume vendor to carry your safety evidence.

How Acumen helps

Acumen runs clinician-graded medical model evaluation for healthcare AI teams and frontier labs. We match verified clinicians to the specialty under test, build physician-written rubrics, grade single answers and multi-step agent runs, adjudicate every disagreement and report agreement. Sets stay held out and ship with documentation for regulatory and health-system review. For the method behind it, read how to evaluate a clinical LLM.

Request a sample dataset Schedule a conversation

FAQ

What do medical LLM evaluation companies do?

They supply the clinicians, rubrics and grading process used to test AI models on medical tasks. That includes writing cases with reference answers, scoring model outputs against physician criteria, resolving disagreements between graders and reporting results. Some also build benchmarks or evaluate agents that take actions inside clinical systems, not only models that answer questions.

Can LLM-as-a-judge replace physician graders?

Not yet on its own. A Harvard and Beth Israel Deaconess preprint found off-the-shelf judges often disagreed with physicians, and an npj Digital Medicine study found the best AI jury matched clinicians on 5 of 11 criteria. Model judges are useful for screening at scale once they are calibrated against physician grades on your own tasks.

How many clinicians should grade each item?

Use at least two for anything that affects safety, with a third to adjudicate disagreements. A November 2026 paper by Google researchers notes that two physicians often diverge by 30 to 40 percent or more on a rating. Reporting inter-rater agreement tells you how much to trust the final score.

What should a medical AI evaluation vendor deliver?

Expect the graded items, the rubric for each case, every grader's specialty and credential record, adjudication notes, agreement statistics and a short method document. For regulated products, you also want a description of how the test data was built, de-identified and kept independent from training data.

How is evaluating a clinical agent different from evaluating a chatbot?

An agent reads records, calls tools and takes actions, so you grade the steps and the final state, not only the text. Scale Labs' CliniCARE-Bench, for example, logs every retrieval and policy lookup so the investigation itself is graded. Rerun each case several times, because consistency matters as much as accuracy.

How does Acumen run a medical LLM evaluation?

We scope the use case and risk level, match verified clinicians in the relevant specialties, write case-specific rubrics and grade outputs with adjudication. You receive scores, rationales, agreement statistics and documentation. Most teams start with a pilot set and a quality report, then rerun the same frozen set on every model release.

Related links

Healthcare medical model evaluation

How to evaluate a clinical LLM

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights