Scientific evaluation imagery representing expert review of medicinal chemistry AI.

Scientific AI evaluation · · 8 min read

How to evaluate AI for medicinal chemistry

Acumen Editorial · Updated November 21, 2026

Frontier labs are now publishing chemistry report cards. On November 20, 2026, OpenAI introduced MedChemBench, a set of 99 graded medicinal chemistry problems built from 117 published papers and patents with 761 distinct structures. Its best model earned 84.5 percent of available credit. An earlier OpenAI model in the same table earned 58.2 percent.

Those are big numbers, and drug discovery teams will hear them in every vendor pitch next quarter. The useful part of the release is less the top score than where models still lose credit. That is where your evaluation should start.

What does MedChemBench actually measure?

Each MedChemBench problem gives a model the evidence available at a chosen point in a real research program. The model is then scored on four components: reconstructing molecular structures, reasoning from the supplied evidence, ranking and selecting compounds, and predicting properties. Numerical predictions earn partial credit based on how close they land to measured values.

The split in the results is the story. OpenAI reports that its top model scored 95.4 percent on structure reconstruction but only about half credit on selection and numerical prediction. In other words, models are now good at reading and redrawing chemistry. They are much less reliable at the decisions a project team actually pays for: which compound to make next and what its properties will be.

OpenAI is also direct about the limits. Prior exposure to the published sources cannot be ruled out. Each answer was a single attempt with no tools, feedback or new experiments. And the benchmark tests reasoning from historical evidence, not whether a model can discover a drug or improve lab decisions.

Why can't a public benchmark stand in for your own test?

Three reasons.

Contamination. Public papers and patents are exactly the kind of text models are trained on. A model that has seen the outcome of a series can "predict" it. Your unpublished programs are the only problems you know it has not seen.

Your chemistry is not the average. A public set spans many targets and modalities. Your portfolio might be kinase inhibitors, degraders or macrocycles. Average credit across 99 problems says little about your next series.

Agents add new failure modes. Teams increasingly buy agents that plan, run code and write up results, not single answers. Other 2026 benchmarks show how far that gap still is:

  • DrugDiscoveryBench (Scale Labs and Phylo, June 2026) set 82 expert-curated agentic drug discovery tasks. The top system solved 51.6 percent, and expert hints let systems solve most of the tasks they had missed.
  • Into the Unknown, a Carnegie Mellon-led benchmark of 15 open research problems including computational chemistry, collected 135 submissions from nine frontier agents. None was valid under strict grading, and 114 failed the integrity check. Common failures were papers that did not match the code, numbers that contradicted the logs and claims of work never done. The authors conclude that "none meets the standard of the domain scientist."

An agent that reports a clean SAR table it never computed is worse than one that says it is unsure. Your evaluation has to catch that.

Which medicinal chemistry tasks should you test?

Break the work into tasks that fail differently, and score each on its own.

TaskWhat goes wrongHow to score itWho grades
Structure handlingWrong ring size, stereocenter or connectivityExact match on canonical structure and stereochemistryAutomated check, chemist spot review
SAR reasoningPlausible story the data does not supportRubric: claims tied to supplied measurementsMedicinal chemists, blind
Compound selectionPicks the wrong next compound to makeRank agreement with what the team actually advanced and whyProject chemists
Property predictionConfident potency or ADME numbers far from assayError against measured values, by property and rangeAutomated, against your assay data
Synthesis planningRoutes that cannot be run or ignore available reagentsFeasibility rubricSynthetic chemists
Agent integrityResults, code and logs disagreeTrace audit: every number reproducible from the runComputational chemists

Weight the table by where money is spent. A structure error is easy to catch. A wrong pick from a ranked list can cost a synthesis cycle.

How do you build a private medicinal chemistry evaluation set?

Here is the checklist we use:

  • Pick closed programs. Use series your team finished or stopped, with real assay results and recorded decisions. Keep them unpublished.
  • Freeze the evidence. Give the model only what the team knew at each decision point, as MedChemBench does with public data.
  • Write reference answers. Chemists record the compound advanced, the rationale and the measured outcome.
  • Include the hard cases. Activity cliffs, unexpected metabolic liabilities, stereoisomers with different activity and series that were killed.
  • Set the conditions. Test both single-answer and tool-using agent modes, and record tools, attempts and effort settings.
  • Grade blind and adjudicate. At least two medicinal chemists per reasoning item, disagreements resolved, agreement reported.
  • Audit agent traces. Check that every reported number appears in the logs and that the code produces it.
  • Measure confidence. Track whether the model says it is unsure when the evidence is thin.
  • Rerun on every change. New model version, new tool or new target class means a new run.

The FDA and EMA principles of good AI practice in drug development, published jointly in January 2026, ask for multidisciplinary expertise, data governance and documentation across the AI life cycle. A written evaluation plan with expert graders and recorded results meets that expectation early, before a model touches a decision that will appear in a submission.

What should you ask an AI vendor for medicinal chemistry?

Ask for evidence, not a leaderboard:

1. Which benchmark results are public data, and which come from unpublished programs?

2. How do you rule out training-data exposure for the problems you report?

3. What is the score on compound selection and property prediction, reported apart from structure tasks?

4. Who wrote and graded the reference answers, and what are their roles and experience?

5. When the tool runs as an agent, can we audit every number against its code and logs?

6. How does the tool signal low confidence, and how often is it right when it does?

7. Can we run our own private evaluation set before purchase and after every update?

A 2026 survey of autonomous research agents by Ding and colleagues found that 83 percent of the 24 systems coded released code, but only 38 percent released seeds or execution traces and only 38 percent reported any novelty check. Ask for the traces.

How Acumen helps

We build the evaluation data behind scientific AI. Our life sciences model evaluation service supplies private evaluation sets written and graded by medicinal chemists, computational chemists, DMPK scientists and synthetic chemists. That includes frozen-evidence problems, reference picks with rationale, reasoning rubrics, agent trace audits and regression runs on every release. Every item is reviewed by experts and adjudicated.

Use it to test a vendor's model, to compare agents before a pilot, or to tune your own system. For how public biology benchmarks fit in, see our analysis of BioBasicsBench.

Request a sample dataset Schedule a conversation

FAQ

What is MedChemBench?

MedChemBench is a medicinal chemistry benchmark OpenAI introduced on November 20, 2026. It has 99 graded problems built from 117 published papers and patents with 761 structures. Models are scored on structure reconstruction, scientific reasoning, compound ranking and selection, and property prediction, with partial credit for numerical predictions close to measured values.

How well do AI models do on medicinal chemistry tasks?

Well on structures, less well on decisions. OpenAI reports its top model scored 95.4 percent on structure reconstruction but about half credit on compound selection and numerical property prediction. Its best overall result was 84.5 percent of available credit, from single attempts without tools, feedback or new experiments.

Why isn't a public chemistry benchmark enough?

Public papers and patents may already be in a model's training data, so a strong score can reflect memory rather than reasoning. Public sets also average across many targets. Only problems built from your own unpublished programs show how a model handles your chemistry, your assays and the decisions your team actually makes.

How do you test an AI agent for drug discovery?

Score its answers and audit its work. Check that every number in its report appears in its logs and that its code reproduces it. A 2026 Carnegie Mellon-led benchmark found none of 135 agent submissions valid under strict grading, with failures including numbers that contradicted logs and claims of work never done.

Who should grade medicinal chemistry AI outputs?

Match graders to the task. Medicinal chemists grade SAR reasoning and compound selection. Synthetic chemists judge route feasibility. DMPK and assay scientists check property predictions against measured data. Computational chemists audit agent code and traces. Use at least two graders per reasoning item and report their agreement alongside the score.

What should be in a medicinal chemistry evaluation set?

Closed, unpublished programs with real assay results and recorded decisions. Freeze the evidence at each decision point, record the compound the team advanced and why, and include hard cases such as activity cliffs and killed series. Add reasoning rubrics, confidence tracking and a rerun after every model or tool change.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights