Scientific AI evaluation · · 7 min read
How to evaluate AI agents for bioinformatics
Acumen Editorial · Updated November 21, 2026
Bioinformatics is where AI agents look most capable. They write the code, call the tools and return a tidy results table. That is also why they are hard to evaluate. A pipeline that runs is not the same as an analysis that is right.
The evidence is getting clearer. OpenAI's GeneBench-Pro, published in June, tests the decisions that shape an analysis, and frontier models solve fewer than a third of its problems. The ICML 2026 BioAgent Bench paper finds agents usually produce the requested files, then fail when inputs are corrupted or padded with decoys. And new open-weight science models keep arriving. Shanghai AI Laboratory released Sage S2.1 Flash on Hugging Face this week, a 321-billion-parameter model that reads protein sequences and 3D structures.
More capable models raise the stakes for the evaluation. Here is how we would run one.
What do bioinformatics benchmarks show so far?
Three public benchmarks tell a consistent story: agents can execute, but they still struggle to decide.
| Benchmark | What it tests | How it was built | Headline result |
|---|---|---|---|
| BixBench (FutureHouse, March 2025) | Open-ended analysis of real datasets | 53 analysis capsules, 296 questions, reviewed by bioinformatics experts | Best model 17% on open answers |
| BioAgent Bench (ICML 2026) | End-to-end pipelines: RNA-seq, variant calling, metagenomics | Curated tasks with output artifacts, scored by an LLM grader | Agents often finish pipelines; perturbations expose step-level failures |
| GeneBench-Pro (OpenAI, June 2026) | Judgment across genomics, quantitative biology and translational medicine | 129 questions on simulated data with known answers; outside experts reviewed 82 | GPT-5.6 Sol 28.7%, or 31.5% in Pro mode |
FutureHouse concluded from BixBench that "fully autonomous bioinformatics research remains out of reach." The newer results show fast progress. OpenAI says the best model on GeneBench's first version scored under 5 percent. But the gap that remains is the expensive one. Reviewers estimated each GeneBench-Pro problem takes an expert 20 to 40 hours.
Why doesn't a working pipeline prove the analysis is right?
Because most bioinformatics errors are silent. A misassigned sample, the wrong reference build or a batch effect left in the data still produces a valid output file. The code runs. The plots look normal. The conclusion is wrong.
BioAgent Bench makes this point directly. Its authors added corrupted inputs, decoy files and bloated prompts. Agents that had chosen the right high-level pipeline still made step-level reasoning errors. The paper also notes that this work often involves sensitive patient data or unpublished intellectual property. That is a reason many teams want local, auditable agents in the first place.
GeneBench-Pro targets the same weakness from another angle. It asks whether the data can support the question at all, how early diagnostics should change the model, and when to revise the plan. OpenAI describes this as research taste. It is the part of the job a senior bioinformatician spends the most time on, and the part a pass rate on "did the pipeline complete" never sees.
Why can't public benchmarks stand in for your own test?
Public benchmarks are useful for screening. They are a weak basis for a deployment decision, for four reasons.
First, they are not your data. Your assays, organisms, sequencing platforms and quality issues differ from any public set.
Second, some are synthetic. GeneBench-Pro uses simulated data so it can grade against known answers. OpenAI notes that this trades some real-world messiness for control.
Third, grading varies. BioAgent Bench relies on an LLM grader. That scales well, but it is not a bioinformatician checking whether the normalization choice was defensible.
Fourth, vendor scorecards are selective. The Sage S2.1 Flash model card reports 34 developer-run benchmarks across coding, productivity, tool use, research and computer use. None of them is a biology analysis benchmark, though the model is built to read scientific inputs.
Data is also becoming a competitive asset. In the Merck and Isomorphic Labs collaboration reported by STAT on November 19, Merck contributes its existing wet-lab data. Your own data deserves the same respect in evaluation: it is the only test set that reflects your science.
What should a bioinformatics agent evaluation include?
We use this checklist when scoping an evaluation with a research informatics team.
- Task set from your own work. Twenty to fifty real analyses your team has already completed, with the final answer and the key decisions recorded.
- Decision points, not just outputs. For each task, list the calls that matter: QC thresholds, reference genome, normalization, covariates, multiple-testing correction, whether to stop.
- Planted faults. Swapped sample labels, a wrong reference build, a hidden batch effect, a decoy file, an underpowered design. Score whether the agent notices.
- "Cannot conclude" cases. Include tasks where the right answer is that the data cannot support the question.
- Trace audit. Check that every number in the agent's report appears in its logs and that the code reproduces it.
- Expert grading with agreement. Two bioinformaticians per task on a written rubric, with agreement reported next to the score.
- Cost and time. Record compute, tokens and wall-clock time per task next to a human baseline.
- Rerun on every change. Freeze the set and rerun it whenever the model, harness or tools change.
This also fits the direction regulators have set. The FDA and EMA's joint principles for AI in drug development, issued in January 2026, call for a defined context of use, risk-based validation and documented data governance. A frozen, expert-graded task set gives you all three.
Who should grade bioinformatics agent outputs?
Match the grader to the decision. Computational biologists who run the same analyses grade pipeline and parameter choices. Statisticians check study design, power and multiple-testing calls. Wet-lab scientists in the relevant area judge whether a conclusion makes biological sense. A clinical geneticist should review anything touching variant interpretation.
LLM judges can help triage at scale. They should not have the last word on the hard cases. In medical evaluation, the research we have covered finds LLM judges miss incomplete answers and favor their own outputs. There is no reason to expect bioinformatics to be different.
What should you ask an AI vendor for bioinformatics?
- Which public benchmarks did you run, and what were the settings, attempts and tools allowed?
- How does the agent behave on corrupted inputs and decoy files?
- Does it ever say the data cannot answer the question? Show examples.
- Can we see full traces, and do reported numbers reconcile with the logs?
- Can it run locally or inside our environment for patient-level or unpublished data?
- Who on your side has graded its analyses, and what are their credentials?
A vendor that only shows completion rates is telling you the pipeline runs. You need to know the analysis is right.
How Acumen helps
Acumen builds private evaluation sets for scientific AI, written and graded by practicing scientists. For bioinformatics agents, that means task sets drawn from analyses like yours, planted faults, decision-level rubrics and graders matched by field: computational biology, statistics, genomics and the relevant wet-lab discipline. We document provenance and grader agreement so the results stand up in a review, and we rerun the set on every release. See how it works on our life sciences model evaluation page, or read what BioBasicsBench means for life sciences AI teams.
FAQ
Can AI agents do bioinformatics analysis on their own?
Not reliably yet. Agents can run standard pipelines such as RNA-seq and variant calling end to end. But on judgment-heavy problems they still fall short. OpenAI's GeneBench-Pro best score was 31.5 percent, and FutureHouse concluded from BixBench that fully autonomous bioinformatics research remains out of reach.
What is GeneBench-Pro?
GeneBench-Pro is an OpenAI benchmark published in June 2026. It has 129 questions across genomics, quantitative biology and translational medicine, built on simulated data with known answers. Outside experts reviewed 82 of them. It tests analysis decisions, and GPT-5.6 Sol scored 28.7 percent, or 31.5 percent in Pro mode.
Why isn't a pipeline completion rate enough?
Most bioinformatics errors are silent. A swapped sample label, the wrong reference build or an uncorrected batch effect still produces valid output files. BioAgent Bench found agents that chose the right pipeline still made step-level errors on corrupted inputs and decoy files. Completion tells you the code ran, not that the analysis is right.
How do you build a bioinformatics evaluation set?
Start with 20 to 50 analyses your team has already completed, with answers and key decisions recorded. Add planted faults and cases where the right answer is that the data cannot support the question. Write a rubric for each decision point, have two bioinformaticians grade every task, and rerun the frozen set on every change.
Who should grade bioinformatics AI outputs?
Computational biologists who run the same analyses grade pipeline and parameter choices. Statisticians check design, power and multiple-testing calls. Wet-lab scientists judge whether conclusions make biological sense, and a clinical geneticist reviews variant interpretation. LLM judges can triage at scale, but experts should decide the hard cases.
Should we run bioinformatics agents locally?
Often, yes. The BioAgent Bench authors note that these workflows frequently involve sensitive patient data or unpublished intellectual property, which is why they focus on cost-effective local agents. Whatever you deploy, evaluate it in the same environment and with the same data access it will have in production.