Life sciences · · 8 min read
How to evaluate autonomous lab AI agents
Acumen Editorial · Updated November 21, 2026
AI agents have started running lab work end to end. This month, SingularityHub and Nature covered AutoDNA, a Shanghai Jiao Tong University system in which LLM agents took one plain-language prompt and wrote a sentence into DNA, then read it back. Agents searched the literature, wrote robot code and drove more than 20 types of equipment. Humans restocked supplies and emptied the trash.
In October, Google Research described VERID, a digital twin that checks an agent's robot commands before any liquid moves. Lab software vendors are shipping protocol-conversion agents. R&D and lab operations leaders are now being asked a practical question: which of these systems can we trust with our instruments, reagents and samples?
Here is how we would answer it.
What can autonomous lab AI agents do today?
More than most teams expect, and less than the headlines suggest. The published results are specific, and so are the failures.
| System | What the agents did | Reported result | Reported limits |
|---|---|---|---|
| AutoDNA (Shanghai Jiao Tong, covered Nov 2026) | Planned and ran DNA synthesis, amplification and sequencing across 20+ instrument types | About 98% PCR yield after iterative tuning, similar to human experts; all 78 strands decoded across 11,382 operations | 14 hardware errors, 6 needing a person; no living cells or animals yet |
| Google Research VERID and aptamer sandbox (Oct 2026) | Designed DNA aptamers over three design-test rounds, with digital-twin dry runs | α-thrombin hit rate rose from 0% to 42.1% by Round 3 at 98.5% specificity | Agents drifted to "sticky" designs; myoglobin specificity fell from 75.0% to 19.1% in one round |
The pattern is consistent. Agents can now execute long, multi-instrument workflows. They also make mistakes a trained scientist would catch, and they optimize for the metric in front of them rather than the result you wanted.
Why isn't a benchmark score enough?
Because lab work fails in the physical world, not on a leaderboard. Google's team names the core problem: agents can produce "syntactically valid code that is physically impossible." Their example is an 8-channel pipette sent into a trough too narrow for it. The code runs. The plan does not.
The second failure is quieter. In the Google aptamer work, agents in one round favored G-quadruplex motifs in 63% of designs. Many bound to off-target proteins and bare beads. The hit rate looked fine until specificity was checked. An agent that chases a proxy metric will find a way to raise it.
The third is tacit knowledge. Stanford's James Zou, commenting on AutoDNA, noted that "experimental success depends on many such details." Evaporation, carryover, dispense height and plate-reader drift rarely appear in a paper's methods section. Google reported uneven dye layering from dispensing too close to the well floor and a plate-reader failure mid-run. A good evaluation tests whether the agent handles these, or at least notices them.
How should you evaluate an autonomous lab agent?
Score the protocol, the physical plan and the judgment separately. A system can pass one and fail another.
| Layer | Question | How to test it | Who grades |
|---|---|---|---|
| Scientific protocol | Is the method right for the goal? | Compare against validated SOPs and published methods for known tasks | Bench scientists in the assay area |
| Physical plan | Can the hardware actually do this? | Dry-run in a digital twin; check volumes, tip counts, labware fit, deck layout | Automation engineers |
| Execution | Did it produce the expected result? | Run known-answer tasks; compare yield, purity, hit rate and specificity with human baselines | Assay owners, with QC data |
| Judgment | Does it pause, ask or stop when it should? | Inject shortages, failed reads and ambiguous instructions | Lab managers and senior scientists |
| Governance | Is every step logged and reviewable? | Audit traces, video, approvals and access controls | QA and biosafety |
Checklist before an agent touches your workcell
- Known-answer tasks. Ten to thirty workflows your team already runs, with established yields and QC ranges.
- Digital-twin dry runs. Every plan simulated first, with invalid commands blocked and logged as failures.
- Protocol review rubric. Scientists grade reagent choice, controls, ordering, timing and safety steps on a written scale, two graders per protocol.
- Proxy-gaming checks. Score the real outcome, not only the agent's target metric. Add counter-screens such as off-target binding or bead-only controls.
- Fault injection. Empty a reagent, fail a read, misplace labware. Score whether the agent pauses and alerts rather than improvising.
- Human-intervention log. Count every fix a person makes. AutoDNA's 6 of 14 is a useful benchmark to ask vendors about.
- Sequence and material screening. Confirm design outputs are screened and that the agent cannot order or run restricted work.
- Full trace. Synchronized commands, instrument state and video for every run.
- Rerun on change. Freeze the task set and rerun when the model, tools, hardware or protocol library changes.
What should you ask an autonomous lab agent vendor?
- On which instruments and labware has the system run, and how many runs failed?
- How many human interventions per run, and what were they?
- Do you dry-run plans in a simulator or digital twin before execution? What does it block?
- How do you stop the agent from optimizing a proxy metric at the expense of the real result?
- Which assay types are out of scope today?
- Who reviewed your protocols, and in which fields are they trained?
- What is logged, and can our QA team audit it under our GxP and Part 11 controls?
Ask for the failure log, not just the case study. The published work is honest about errors. A vendor should be too.
Who should grade an agent's lab work?
Match the grader to the layer. A scientist who runs the assay judges whether the protocol is sound and whether the result means what the agent says it means. An automation engineer judges whether the plan fits the hardware. A lab manager judges the stop-and-ask behavior. For translational work, a pharmacologist or clinician judges whether the experiment answers a question that matters downstream.
LLM judges help with triage, such as flagging missing controls or mismatched volumes. They should not sign off on a protocol that will consume samples. Google's own framing keeps humans in charge of strategy and review: scientists set direction, authorize designs and supervise robot runs.
How does this fit regulatory expectations?
Most autonomous lab work sits in discovery, well before submission. But the data it produces feeds programs that regulators will review. The FDA and EMA guiding principles for AI in drug development, issued in January 2026, call for risk-based performance assessment, data quality and life cycle management. A frozen, expert-graded evaluation set with full traces gives you that record before an agent's output becomes part of a program.
How Acumen helps
Acumen builds private evaluation sets for scientific AI, written and graded by practicing scientists. For autonomous lab agents, that means known-answer workflows drawn from assays like yours, protocol rubrics, fault-injection scenarios and grading by bench scientists matched to the assay, from molecular biology to analytical chemistry. We document provenance and grader agreement and rerun the set on every release. See how it works on our life sciences model evaluation page, or read how to evaluate an AI co-scientist.
FAQ
What are autonomous lab AI agents?
Autonomous lab AI agents are language-model systems that plan experiments, write instrument code and drive lab robots with limited human input. In November 2026, Nature and SingularityHub covered AutoDNA, in which cooperating agents ran DNA synthesis, amplification and sequencing across more than 20 instrument types. People mainly restocked supplies and fixed some hardware errors.
Can AI agents really run a biology lab?
For some workflows, yes. AutoDNA completed 11,382 operations and decoded all 78 DNA strands it wrote, with PCR yields near 98 percent after tuning. It still logged 14 hardware errors, 6 of which needed a person, and it cannot yet handle living cells or animals. Today these systems work best on well-defined molecular workflows.
What is a digital twin for lab automation?
A digital twin is a software model of the lab workcell that tracks robot geometry, volumes, reagents and tip inventory. Google Research's VERID, described in October 2026, uses one to dry-run agent plans and block physically invalid commands before execution. For evaluation, the twin turns impossible plans into logged failures instead of wasted samples.
Why do lab agents fail even when their code runs?
Because valid code can describe an impossible action. Google's team gives the example of an 8-channel pipette sent into a trough too narrow for it. Agents also miss tacit details such as evaporation, carryover and dispense height, and they can chase a proxy metric, as when designs raised hit rates but bound off-target.
How do you test an autonomous lab agent before deployment?
Start with known-answer workflows your team already runs. Dry-run every plan in a digital twin, then compare real results with human baselines on yield, purity and specificity. Inject faults such as empty reagents or failed reads and score whether the agent pauses. Bench scientists should grade protocols on a written rubric.
Who should grade an AI agent's lab protocols?
Scientists who run the assay should grade the protocol and interpret results. Automation engineers should judge whether the plan fits the hardware, and lab managers should judge stop-and-ask behavior. LLM judges can flag missing controls or mismatched volumes for triage, but a person should sign off before any protocol consumes real samples.