Case authoring
physicians write cases from their own practice, with presentation, workup, constraints and a reference answer the model is graded against
We are the healthcare specialist post-training teams add to their data stack. Custom projects, specialty-matched experts, layered QA and documentation that holds up in regulated deployments.
Request a sample datasetSchedule a conversationExpert data for AI labs is training and evaluation data authored, graded or ranked by credentialed domain specialists rather than generalist labelers. In healthcare it covers case authoring, rubric design, preference pairs, reasoning traces and RL environments for clinical workflows. Acumen builds it to spec for post-training teams, with specialty matching, multi-layer QA and full provenance.
Expert data is now a core input to post-training. TechCrunch (Aug 2026) reported one expert-data provider reaching a $500M gross run rate and described lab demand for unique training data as near-bottomless. Labs already work with several providers at once. Healthcare is where those providers get tested hardest.
Horizontal benches cover every domain. Healthcare asks for more: a pediatric nephrologist this week, rubrics that hold up under three-physician adjudication, and pharma context most general vendors don't have. When the bench runs thin, throughput drops and QA falls back to a single reviewer.
We built Acumen for the work generalist benches find hardest.
The questions we hear on every scoping call:
A general pool sends you a physician. Your task needed a hematologist who treats the condition in question. The gap shows up later as noisy preference pairs and rubrics that miss what matters.
We work only in healthcare, life sciences and pharma. Every task is matched by specialty, subspecialty and practice setting.
This enables something a general pool cannot: the right specialist, on schedule.
Evaluation has moved beyond multiple-choice exams. OpenAI's HealthBench grades 5,000 multi-turn health conversations against 48,562 rubric criteria written by 262 physicians, and HealthBench Professional (2026) uses rubrics adjudicated by three or more physicians. Post-training data now has to meet the same bar. We build four kinds:
physicians write cases from their own practice, with presentation, workup, constraints and a reference answer the model is graded against
conversation-specific criteria that score whether the differential is ordered sensibly and whether a guideline actually applies to this patient
specialists compare two model responses to the same prompt and record which is better, and why
patient scenarios, workflow tasks and physician-built graders that produce a reward signal for clinical reasoning
Healthcare post-training moves from best effort to a specialist-built, auditable process.
A 2025 literature survey on HealthBench found that strong average scores can mask dangerous weaknesses in high-stakes situations. Worst cases matter more than averages, and they move every release. The questions don't stop:
We answer them with expert data on a recurring cadence, scoped to each release.
Breadth is easy to buy. Specialty depth is not.
From a general expert pool → a specialty bench built for post-training.
| Acumen | Horizontal expert networks | Crowdsourced annotation | In-house team | |
|---|---|---|---|---|
| Who does the work | Credentialed specialists in healthcare, life sciences and pharma | Vetted experts across many domains | Generalist workforce | Your staff or contractors |
| Rare subspecialty staffing | Matched per task | Varies by domain depth | Not available | Slow to hire |
| QA layers | Author, reviewer, adjudicator, batch sampling | Varies by project | Spot checks | Varies |
| Rubric and RL environment design | Built and adjudicated by physicians | Available, domain depth varies | Not suited | Your team's capacity |
| Pharma and commercial context | Core expertise | Limited | None | Depends on hires |
| Regulated-deployment documentation | Full provenance per item | Varies | Rarely | Your burden |
| Role in a multi-vendor stack | Healthcare specialist | Generalist coverage | High-volume simple tasks | Core IP and oversight |
Regulators now ask about training data. FDA's January 2025 draft guidance on AI-enabled device software asks sponsors to describe data collection, cleaning and processing, test-data independence and bias. In the EU, general-purpose AI model obligations under the AI Act have applied since August 2, 2025. Your enterprise customers inherit those questions, then pass them to you.
Every Acumen item carries its provenance: who authored and reviewed it, their specialty and how disagreements were resolved. We package it with a data card your customers' procurement, legal and clinical reviewers can read directly.
The market has moved from cheap generic labeling to expert review. Reasoning models created demand for doctors, lawyers and PhDs (Sacra). In healthcare, the next step is specialists who know the specialty, the workflow and the regulations.
Acumen is the healthcare specialist in your expert data stack, for training, evaluating and deploying AI where the stakes are clinical.
Training and evaluation data authored, graded or ranked by credentialed specialists instead of generalist labelers. For labs entering healthcare, that means physician-written cases with reference answers, scoring rubrics, preference pairs, reasoning traces and RL environments for clinical workflows. Acumen builds each project to your spec, across healthcare, life sciences and pharma, with specialty matching and full provenance.
Yes. Because we work only in healthcare, life sciences and pharma, we recruit inside networks we already know. If a subspecialty is genuinely scarce, we tell you on the scoping call, not after kickoff.
A second specialist reviews the author's work, disagreements go to a senior adjudicator in the same specialty, and we sample delivered batches against your rubric. You get inter-reviewer agreement per batch and a log of how each disagreement was resolved. Items that fail adjudication are rewritten, not shipped.
Yes. We design environments around real clinical workflows: patient scenarios with realistic constraints, tasks the model must complete, and physician-written rubrics or reference answers that produce a reward signal. Typical tasks include building a differential, choosing a workup, drafting documentation or answering a care consult. Specialists stress-test each environment so the grader rewards sound reasoning, not shortcuts.
We price to the shape of the work. Fixed deliverables, such as an evaluation set or rubric library, are quoted per dataset. Repeatable work, such as preference pairs or grading, is priced per task. Open-ended work, such as red teaming or environment design, is priced per expert hour. You get a written quote after the scoping call, before any work starts.
Yes. TechCrunch reports that labs commonly work with several expert data providers at once. We slot in as the healthcare specialist. We match your schemas, rubric conventions and delivery format, and take on the specialty, pharma and regulated-deployment work that general benches find hardest to staff.
Physicians write conversation-specific criteria: what a good answer must include, what it must avoid and how much each point weighs. OpenAI's HealthBench uses this approach, with 48,562 criteria written by 262 physicians. We pilot each rubric on sample outputs and check that reviewers apply it consistently before it enters your pipeline.
Every delivery includes provenance: who authored and reviewed each item, their specialty and credentials, and how disagreements were resolved. It also includes a data card describing collection and processing. FDA's January 2025 draft guidance on AI-enabled devices asks sponsors to describe these points, including test-data independence. Your customers' procurement, legal and clinical teams can review it directly.
Last updated November 2026