Life Sciences · · 8 min read
How to evaluate an AI co-scientist
Acumen Editorial · Updated November 21, 2026
AI co-scientists have moved from demo to product. Google published its Co-Scientist research in Nature in May, a multi-agent system that generates, debates and ranks hypotheses. Lab software vendors now ship agents that turn protocols into automated runs. R&D teams are being asked to decide which of these systems deserve lab time.
That decision is harder than it looks. A hypothesis is cheap to write and expensive to test. Google DeepMind's own policy team put it plainly in July: "AI agents are conjecture machines." Generating ideas is no longer the bottleneck. Choosing which ones to test is.
Here is how we would evaluate a co-scientist before it shapes a research budget.
What does the evidence say about AI-generated hypotheses?
The picture is mixed, and the method behind each result matters.
| Study | What it measured | How it was scored | Headline finding |
|---|---|---|---|
| Google Co-Scientist (Nature, May 2026) | Hypotheses for open biomedical problems | Collaborating scientists, then lab tests | In a liver fibrosis case, one candidate blocked 91% of a scarring-linked response in lab tests |
| MechHypoBench (arXiv 2610.05197, Oct 2026) | Turning real data into mechanistic hypotheses | Predictions checked against withheld conditions | Best agent 64.9% on easy tasks and 50.8% on hard, vs. 87.6% and 82.9% for reference mechanisms |
| Bao, Evans et al. (arXiv 2606.08251, 2026) | Novelty, feasibility and likely truth of LLM ideas | 6,749 scientists rated ideas drawn from their own papers | No model spontaneously proposed a null hypothesis |
The wins are real. Google DeepMind reports that Co-Scientist returned a top-ranked hypothesis for Imperial College microbiologist José Penadés that matched a mechanism his team had spent years working to prove. In the liver fibrosis work with Stanford's Gary Peltz, two of its three repurposing picks blocked fibrosis in lab assays.
But the controlled benchmarks show how far general performance still has to go. MechHypoBench, built from 244 papers and 150 datasets across 14 fields, found a "substantial gap" between generated hypotheses and the true mechanisms. Even the best configuration trailed the reference answers by 23 to 32 points.
Why can't you judge a hypothesis by reading it?
Because fluency and correctness come apart. A co-scientist can cite the right pathways, use the right vocabulary and still propose a mechanism the data does not support.
MechHypoBench makes two useful points here. First, its shuffled control, which pairs tasks with mismatched mechanisms, still scored 38 to 40 percent. Some apparent accuracy comes from chance and generic biology. Your baseline has to account for that. Second, purpose-built AI-scientist workflows did not consistently beat a general agent running on the same model. The label on the product told you less than the test did.
The Bao and Evans study adds a quieter failure. Non-reasoning models converged on a narrow set of similar ideas, and no model volunteered a null hypothesis. Good scientists propose "no effect" often. A system that never does will push your team toward experiments that confirm rather than test.
What should an AI co-scientist evaluation include?
We use this checklist when scoping an evaluation with an R&D team.
- Questions you have already answered. Fifteen to forty problems from your own programs where the mechanism or result is now known, written as they looked before the answer was in.
- Held-out evidence. Give the system only the data available at the time. Keep the later experiments back and score its predictions against them.
- Predictions, not prose. Require each hypothesis to state a testable consequence: what changes, in which direction, under which condition.
- Null and dead-end cases. Include problems where the honest answer was no effect or not enough data. Score whether the system says so.
- Controls. Run an empty and a shuffled baseline so you know what chance looks like on your set.
- Expert rubric. Domain scientists grade novelty, mechanistic plausibility, feasibility and fit with prior evidence, with two graders per hypothesis and agreement reported.
- Diversity check. Count how many distinct mechanisms the system proposes across runs, not just how good the top one is.
- Citation and evidence audit. Every cited paper exists and says what the hypothesis claims.
- Budget and attempts. Record compute, runs per question and how the top answer was selected.
- Rerun on every change. Freeze the set and rerun it when the model, tools or workflow change.
The last two items matter more than they look. An October 2026 survey of automated-research evaluation (arXiv 2610.11877) found that resource budgets and attempt selection change how results should be read. A system given fifty tries and a human picker is a different product from one given a single try.
Who should grade AI-generated hypotheses?
Match the grader to the question. A scientist who works on the target biology judges whether the mechanism is plausible and new. A biostatistician judges whether the proposed test can actually distinguish the hypothesis from its alternatives. For translational questions, a clinician or pharmacologist judges whether the idea survives contact with patients, dosing and safety.
LLM judges have a place. They can triage hundreds of hypotheses and flag obvious duplicates. But the same survey warns that a judge calibrated for one property may not be valid for another. MechHypoBench, for example, uses an LLM to interpret hypotheses and does not report agreement with human graders. For decisions that commit lab time, experts should grade the shortlist.
What should you ask an AI co-scientist vendor?
- Which hypotheses from your system were tested in a lab, and how many failed?
- How do you score hypotheses before testing, and has that score been checked against outcomes?
- Does the system ever propose a null hypothesis or say the data cannot answer the question? Show examples.
- How many attempts and how much human curation sit behind the results you show us?
- Can we run it on our own held-out problems, inside our environment?
- Who graded its outputs, and in which fields are they trained?
Ask for failures as well as wins. Case studies are selected. A track record includes the misses.
How does this fit regulatory expectations?
Most co-scientist work happens well before any submission. But the outputs feed programs that will be reviewed. The FDA and EMA's joint guiding principles for AI in drug development, issued in January 2026, call for a defined context of use, risk-based performance assessment and documented data governance. A frozen, expert-graded evaluation set gives you the record those principles expect, before the hypothesis turns into a program.
How Acumen helps
Acumen builds private evaluation sets for scientific AI, written and graded by practicing scientists. For AI co-scientists, that means retrospective problem sets drawn from work like yours, held-out evidence, null cases and rubrics for novelty, plausibility and testability. Graders are matched by field, from cell biology and pharmacology to biostatistics. We document provenance and grader agreement and rerun the set on every release. See how it works on our life sciences model evaluation page, or read how to evaluate AI agents for bioinformatics.
FAQ
What is an AI co-scientist?
An AI co-scientist is a system, usually several cooperating language-model agents, that reads literature and data, proposes hypotheses, critiques them and ranks the best. Google's Co-Scientist, published in Nature in May 2026, is the best-known example. These systems are built to support researchers in choosing what to test, not to replace their judgment.
Can AI generate good scientific hypotheses?
Sometimes. Google DeepMind reports cases where Co-Scientist proposed mechanisms and drug candidates that later held up in the lab. But controlled tests are more sobering. On MechHypoBench, the best agent scored 50.8 percent on hard tasks, against 82.9 percent for reference mechanisms. Performance depends heavily on the problem, the data and the setup.
What is MechHypoBench?
MechHypoBench is a benchmark posted to arXiv in October 2026 that tests whether AI agents can turn real experimental data into mechanistic hypotheses. It has 300 tasks built from 244 papers and 150 datasets across 14 fields. Hypotheses are scored by whether their predicted consequences match withheld conditions, rather than by how they read.
How do you test an AI co-scientist on your own data?
Pick problems your team has already solved and write them up as they looked beforehand. Give the system only the evidence available then. Ask for testable predictions, score them against what you later learned, and add null cases and shuffled controls. Two domain scientists should grade each hypothesis on a written rubric.
Why does it matter if an AI never proposes a null hypothesis?
Because good science tests ideas rather than confirms them. A 2026 study in which 6,749 scientists rated LLM-generated ideas found no model spontaneously proposed a null hypothesis. A system that always predicts an effect will steer teams toward confirmatory experiments and make it harder to kill weak ideas early.
Can an LLM judge grade AI-generated hypotheses?
It can help with triage, such as removing duplicates and flagging weak evidence. It should not make the final call. An October 2026 survey of automated-research evaluation warns that a judge calibrated for one property may not be valid for another. For hypotheses that will consume lab time, domain experts should grade the shortlist.