Life sciences · · 8 min read
How to evaluate AI variant effect prediction models
Acumen Editorial · Updated November 22, 2026
Variant effect prediction models now shape which variants a team follows up, which targets move forward and which hypotheses reach the bench. The models are getting better fast. The tests most buyers rely on have not kept pace. For computational biology leads, functional genomics teams, target discovery groups and the AI teams building these tools, the question is simple: does a published benchmark tell you how the model will perform on your variants, in your cell types? Usually it does not.
Why are variant effect prediction models moving now?
Because they now predict effects cell type by cell type. On November 11, 2026, Google DeepMind released AlphaGenome 2. It takes a one-million-letter DNA sequence plus a target tissue, cell type or cell state, and predicts gene expression, chromatin accessibility, histone modifications and DNA methylation. DeepMind says it was trained on more than 100 million single-cell profiles across 68 tissues.
The reported gains are large. On the TenK10K dataset of 28 primary immune cell types, DeepMind reports AlphaGenome 2 identified causal variants 30% of the time, versus 16% for the original AlphaGenome fine-tuned on single-cell data and other specialized tools. It identified the direction of gene effects 43% of the time, versus 23%.
The first AlphaGenome, published in Nature in January 2026, matched or beat existing models on 25 of 26 variant effect evaluations. AlphaGenome 2 is available for non-commercial research through an API, with commercial access in private preview on Google Cloud.
DeepMind is also clear about scope. Both models are for research use only and "have not been validated or approved for clinical or diagnostic purposes." That puts the burden of validation on the team using them.
Why don't published benchmarks settle the question?
Three reasons.
Circularity. Many predictors are trained or tuned on the same clinical databases used to test them. A model can score well by recognizing variants, or close relatives of them, that it has already seen. Livesey and Marsh, in Molecular Systems Biology (2023), benchmarked 55 predictors against deep mutational scanning data for 26 human proteins to keep circularity to a minimum. Unsupervised models did well, and the protein language model ESM-1v ranked first overall. Rankings change when the labels change.
Thresholds. A score is not a decision. In a 2022 American Journal of Human Genetics study, the ClinGen Sequence Variant Interpretation Working Group calibrated 13 tools for missense pathogenicity. Developer default thresholds for SIFT, PolyPhen-2 and CADD did not meet the supporting-evidence standard for computational evidence. The group recommends calibrated thresholds and a single tool chosen before anyone sees its predictions.
Scope. A headline result comes from specific cell types, variant classes and assays. The TenK10K result covers immune cells. If your program is in neurons, hepatocytes or tumor tissue, you have to test there.
What should a variant effect prediction evaluation test?
Score each layer separately. One overall correlation hides which failure you have.
| Layer | What it catches | How to score it |
|---|---|---|
| Held-out split | Leakage from training data | Hold out by gene, region or chromosome, not by random variant |
| Independent labels | Circular, database-driven scores | Functional assays, deep mutational scans or new fine-mapping results |
| Cell-type fit | Results that do not transfer | Separate sets for each tissue and cell state you work in |
| Variant class | Uneven performance | Score missense, splice, promoter, enhancer and structural variants apart |
| Direction and size | Right variant, wrong effect | Check predicted direction and magnitude against measured change |
| Calibration | Scores read as certainty | Map scores to evidence levels on a held-out calibration set |
| Expert review | Plausible but wrong calls | Genetics experts grade prioritized lists and stated mechanisms |
| Stability | Shifts across versions | Rerun the same set on every model, data or pipeline change |
How do you test cell-type-specific predictions?
Treat each cell type as its own test. A model that is strong in immune cells can be weak in the tissue your program depends on, and an average across tissues will hide it.
Start with the cell types where you have measured data: single-cell expression, chromatin accessibility or reporter assays from your own experiments or from studies the model was not trained on. Score whether the model ranks the right variants, predicts the right direction of change and keeps effects in the right cell type rather than spreading them everywhere.
Then test the limits on purpose. Ask the model about cell states it was not trained on and check how much its performance drops. Compare it with the simpler predictor you already use. If the new model is not clearly better on your cells, the headline benchmark does not matter for your program.
Checklist for building a variant effect test set
- Define the decision first: variant prioritization, target discovery, fine-mapping follow-up or assay design. The pass bar follows from it.
- List the cell types, tissues and variant classes that matter to your program.
- Check each candidate test variant against the model's documented training sources. Remove overlaps and near neighbors.
- Hold out by gene or genomic region so the model cannot score well by memory.
- Use labels from functional assays or new experiments where you can, not only clinical databases.
- Enrich for the hard cases: noncoding variants, rare cell states, small effects and conflicting evidence.
- Calibrate thresholds on a separate set and record them before running the final test.
- Have qualified experts double-grade ambiguous labels and adjudicate disagreements. Report agreement.
- Keep the set private and rerun it on every model, data or pipeline change.
- Log versions, splits, thresholds and grades for review.
Where do experts fit in a variant effect evaluation?
At the points where a model's output becomes a decision. Benchmarks measure correlation with a label. Teams act on ranked lists, mechanisms and follow-up plans.
Expert review covers what an automated metric misses:
- Label quality. Functional genomics scientists and molecular geneticists decide whether an assay result is a fair label for the question, and flag variants with conflicting evidence.
- Prioritized lists. Statistical geneticists and target discovery scientists judge whether the top-ranked variants are biologically credible or artifacts of the model.
- Stated mechanisms. When a model or an AI assistant explains a predicted effect, an expert checks whether the mechanism holds up for that gene and cell type.
- Clinical boundaries. If any output could reach patient care, clinical geneticists confirm it is framed as research evidence, not a diagnosis.
What should you ask a variant effect evaluation vendor?
- Which experts write and grade the test set, and how are their credentials verified?
- How do you check test variants against the model's training data?
- Do you hold out by gene or region, or by random variant?
- Which labels do you use, and how independent are they of public clinical databases?
- Do you report results by cell type and variant class, not only overall?
- Can we keep the set private and rerun it ourselves?
A vendor that reports one correlation across all variants has not answered these.
How Acumen helps
We build private evaluation sets for variant effect prediction and genomics AI. Verified functional genomics scientists, molecular and statistical geneticists and clinical geneticists curate held-out variants, check labels, grade prioritized lists and stated mechanisms, and adjudicate disagreements. We split sets by cell type and variant class, document provenance and rerun them on every release.
See our scientific AI model evaluation service, or read our guide to evaluating AI agents for bioinformatics.
FAQ
How do you evaluate a variant effect prediction model?
Test it on variants it could not have seen in training, held out by gene or genomic region. Use independent labels such as functional assays or deep mutational scans, calibrate score thresholds on a separate set, and report results by cell type and variant class. Have genetics experts review ambiguous labels, ranked lists and stated mechanisms.
What is data circularity in variant effect prediction?
Circularity happens when a predictor is trained or tuned on the same databases used to test it, so it scores well by recognizing variants it has effectively seen. Benchmarks built on independent data, such as deep mutational scanning, reduce it. Livesey and Marsh used that approach in 2023 to compare 55 predictors across 26 human proteins.
Are developer default score thresholds reliable?
Not always. In a 2022 ClinGen calibration study of 13 missense tools, developer default thresholds for SIFT, PolyPhen-2 and CADD did not meet the supporting-evidence standard for computational evidence. Calibrate thresholds on a held-out set for your use, record them before the final test, and map scores to evidence levels rather than reading them as certainty.
What does AlphaGenome 2 change for genomics teams?
It predicts how variants affect specific cell types and states. DeepMind reports 30% causal variant identification versus 16% for prior approaches on 28 immune cell types. It is for research use only and not validated for clinical or diagnostic use, so teams still need to test it on their own cell types, variants and decisions.
Can AlphaGenome or similar models be used for clinical variant interpretation?
Not as released. DeepMind states AlphaGenome and AlphaGenome 2 are for research use only and have not been validated or approved for clinical or diagnostic purposes. Clinical variant classification follows established evidence frameworks, and any computational evidence needs calibrated thresholds. Treat these models as research tools and keep clinical geneticists in any patient-facing decision.
Who should grade a variant effect evaluation set?
Functional genomics scientists and molecular geneticists for label quality, statistical geneticists and target discovery scientists for prioritized lists, and clinical geneticists where outputs could reach patient care. Verify credentials, double-grade ambiguous labels, adjudicate disagreements and report inter-rater agreement with the results. Match graders to the tissues and variant classes in the set.