Private test sets
realistic cases built for your product, kept out of training data
Acumen builds private, physician-written evaluations for clinical models and agents, and reruns them every release.
Request a sample evaluationSchedule a conversationMedical LLM evaluation measures how a model performs on real clinical tasks, scored against criteria written by physicians. Strong evaluations use realistic cases, specialty-specific rubrics, multiple graders and held-out test sets. Acumen builds them around your product and your patients, then reruns them with every release.
Exam-style benchmarks tell you a model knows medicine. They don't tell you whether your product gives a safe answer to your users. OpenAI's HealthBench moved the field to physician-written rubrics for exactly this reason: 262 physicians wrote criteria for 5,000 realistic conversations.
Averages hide the misses that matter. A literature review of HealthBench results found that strong average scores can mask dangerous weaknesses in high-stakes situations. In healthcare, the worst answer is the one that counts.
Acumen builds the evaluation your product actually needs.
Internal clinician spot checks are slow, small and hard to repeat. Results depend on who happened to review that week.
Acumen recruits specialists by domain, writes rubrics with them, and has every item graded by multiple physicians with disagreements adjudicated.
This enables something spot checks cannot: evaluation you can repeat and defend.
realistic cases built for your product, kept out of training data
specific, gradable criteria per case, not a single score
failures sorted by severity, so the dangerous ones surface first
the same evaluation rerun on every model or prompt change
Launches move from hope to a measured, physician-graded decision.
Models change. Prompts change. Guidelines change. FDA's lifecycle approach to AI-enabled devices and its guidance on predetermined change control plans both assume performance is checked after launch, not just before. The questions repeat every release:
Acumen reruns your evaluation on a set cadence and reports what changed.
Benchmarks rank models. Evaluations decide launches.
From one-off reviews → a living evaluation system.
| Acumen evaluation | Public benchmarks | Internal spot checks | |
|---|---|---|---|
| Built for your product | Yes | No | Partly |
| Graded by | Physicians by specialty, multiple per item | Fixed rubric or answer key | Whoever is available |
| Worst-case visibility | Severity-ranked failures | Average scores | Anecdotal |
| Kept out of training data | Private, held-out | Often public | Varies |
| Repeatable every release | Yes | Yes, but generic | Hard |
| Useful for buyers and regulators | Documented method and audit trail | Limited | Rarely |
Every score traces back to a case, a rubric, the physicians who graded it and how disagreements were resolved.
Acumen packages the method and results so security, clinical and regulatory reviewers can see exactly how the model was tested.
Models are getting better fast. The question is no longer whether a model knows medicine. It's whether your product is safe and useful for the people who rely on it.
Acumen is the physician-graded evaluation layer for launching, improving and proving clinical AI.
Build realistic cases from your product's use cases, write case-specific rubrics with physicians, have multiple physicians grade each output, and report severity-ranked failures alongside overall scores. Keep the test set private and rerun it on every release.
Public benchmarks are useful for comparing general models, but they aren't built around your product, users or workflows, and some are already in training data. A private evaluation tells you whether your product is ready.
It depends on risk and specialty. We use multiple graders per item and adjudicate disagreements, and we agree the reviewer count with you during scoping.
Test sets are private, held out and delivered only for evaluation. We document provenance and keep evaluation material separate from any training data we produce.
Yes. We evaluate multi-step workflows, including when an agent asks a follow-up question, escalates to a clinician or refuses, against criteria physicians define.
We recruit by specialty for each project, including primary care, emergency medicine, cardiology, oncology and other specialties, based on your product's use cases.
On every model, prompt or retrieval change that could affect clinical behavior, and on a regular cadence as guidelines change.
Scores by task and specialty, severity-ranked failures with examples, inter-rater agreement, the rubric set and a rerunnable test set.
Last updated November 2026 · Reviewed by [Name, credential]