Clinical research imagery representing expert review of healthcare AI agent skills, clinical protocols and evaluation cases.

Healthcare & Life Sciences · · 7 min read

How to evaluate AI agent skills for healthcare and life sciences

Acumen Editorial · Updated November 21, 2026

Health AI has a new unit of work. Instead of fine-tuning a model or writing a longer system prompt, teams now package procedure as skills: short instruction files, sometimes with scripts, that an agent loads when a task calls for them.

That shift moved fast at HLTH this week. For clinical AI leads and R&D teams, it raises one practical question. Who checks the skill?

What are AI agent skills, and why do they matter now?

A skill is a packaged procedure. It tells an agent what knowledge applies, which steps to follow and when to stop. Skills can be loaded by any assistant that supports them, often through the Model Context Protocol (MCP).

On November 17, 2026, b.well Connected Health launched health.skills, a library of expert-written skills that teach AI assistants to answer specific health questions by following protocols and safety checks. According to the release:

  • Each skill is written in plain English and covers the clinical knowledge, the steps and the checks that must pass before an answer is given.
  • Core skills draw on CDC vaccine recommendations, USPSTF screening guidance, HEDIS care-gap criteria and the AHA PREVENT model.
  • Skills run against the person's live record in US Core FHIR R4 and stop if required information is missing.
  • Core skills check for guidance from the patient's own doctor before applying national guidelines.
  • Organizations can write and update their own skills, and b.well plans a skills and agents marketplace.

That is a sound design. It also means a growing share of clinical behavior will live in files written by many hands, outside the model and outside its vendor's testing.

Do skills actually make agents better?

Curated ones do, on average. SkillsBench, a benchmark from Li and colleagues accepted at NeurIPS 2026, tests 87 tasks across 8 domains with deterministic verifiers and 18 model-harness configurations.

  • Curated skills raised the average pass rate from 33.9% to 50.5%.
  • Gains by configuration ranged from 4.1 to 25.7 percentage points.
  • Focused skills with three modules or fewer beat large, exhaustive bundles.
  • Smaller models with skills could match larger models without them.

Read the range, not the average. A skill that helps one model a lot may help another very little. And a skill library that keeps growing may get worse, not better, if it trades focus for coverage.

What can go wrong with a clinical skill?

Two kinds of failure. The skill can be clinically wrong, or it can be unsafe as software.

On the software side, Liu and colleagues analyzed 31,132 skills from two public marketplaces in January 2026. They found 26.1% contained at least one vulnerability, across prompt injection, data exfiltration, privilege escalation and supply-chain risks. 5.2% showed high-severity patterns that strongly suggest malicious intent. Skills that bundled executable scripts were 2.12 times more likely to be vulnerable than instruction-only skills.

On the clinical side, scale is outrunning review. FreedomIntelligence's open-source OpenClaw Medical Skills repository lists 869 skills, including 119 medical and clinical skills and 239 bioinformatics skills, aggregated from more than a dozen open-source repositories. In the README we reviewed, we did not find a description of clinical validation for individual skills. Some skill descriptions cite HIPAA, FDA or ICH-GCP. A citation is not evidence.

Clinical failure modes we test for:

  • Outdated guidance, such as a screening interval that changed after the skill was written
  • Wrong thresholds or units copied from a source table
  • Missing exclusions, contraindications or special populations
  • A stop condition that never fires, so the agent answers from incomplete data
  • National guidance applied over a treating clinician's documented plan
  • Two skills that conflict, with no rule for which wins

How should you evaluate an AI agent skill?

Treat each skill like a clinical protocol and a piece of code at the same time. Score the layers separately.

LayerWhat it catchesHow to score it
Source fidelityWrong or outdated guidance, thresholds, unitsClinicians trace every rule to its cited source and version date
Decision logicMissing exclusions, wrong branches, edge populationsSpecialist-written test cases with known correct answers, including traps
Stop conditionsAnswering from incomplete or conflicting dataCases with missing labs, stale records and conflicting notes; score refusals
Lift over baselineSkills that add nothing or hurtSame cases with and without the skill, per model you run
InteractionConflicts between skills and with clinician plansCases that trigger two or more skills at once
SecurityInjection, exfiltration, excess permissionsCode and permission review, especially for skills with scripts
DriftSkills that age badlyRe-run on every guideline update, model change and skill edit

The first three rows need clinicians and scientists, not a scanner. The fourth row is where most teams skip a step: if you never test the model without the skill, you cannot say the skill earns its place.

What belongs in a skill review checklist?

  • Every clinical rule cites a source, a version and a review date
  • A named, credentialed reviewer signed off, with specialty matched to the skill
  • Test cases cover common presentations, edge populations and planted traps
  • Stop conditions are tested with missing, stale and conflicting data
  • The skill is scored with and without loading, on each model in production
  • Scripts, network calls and data access are reviewed and permissioned
  • Conflicts with other skills and with clinician-documented plans are tested
  • A re-test is triggered by guideline changes, model updates and skill edits
  • Results are logged so you can show what was tested, when and by whom

Life sciences teams should apply the same list to bioinformatics and lab workflow skills. A wrong parameter default in a pipeline skill does not harm a patient directly. It can still corrupt a dataset that a decision rests on.

What should you ask a skills vendor or marketplace?

  1. Who wrote each clinical skill, and what are their credentials?
  2. Which sources and versions does each skill encode, and how do you track updates?
  3. What test cases does each skill pass, and who wrote the expected answers?
  4. Have you measured lift over the same model without the skill?
  5. How do you vet third-party skills for security before they reach a marketplace?
  6. Can we run our own private test set against every skill release?

Ask for the test set, not the description. A vendor that has tested its skills can show the cases.

Amazon framed the bar well this week when it expanded patient-facing agents in Amazon Connect Health, pairing deterministic handoff rules with ongoing human clinical review. As AWS's Kimbrough told Newsweek, "Trust and safety are the product." Skills deserve the same standard.

How Acumen helps

We build private evaluation sets for AI agent skills in healthcare and life sciences. Practicing clinicians, pharmacists and scientists trace each rule to its source, write test cases with known answers, plant the edge cases that matter and grade agent behavior with and without the skill. We document reviewer credentials and provenance for audit, and re-run the set when guidelines, models or skills change.

See our medical model evaluation service, or read how to evaluate EHR AI agents.

Request a sample dataset Schedule a conversation

FAQ

What is an AI agent skill?

An AI agent skill is a packaged procedure an agent loads when a task needs it. It usually combines plain-language instructions with references and sometimes scripts. In healthcare, a skill can encode a guideline, the steps to apply it and the checks that must pass first. Skills are often shared through the Model Context Protocol, so many assistants can use the same file.

Do agent skills improve AI performance?

Curated skills usually do. In SkillsBench, a NeurIPS 2026 benchmark of 87 tasks across 8 domains, curated skills raised the average pass rate from 33.9% to 50.5%. Gains varied widely, from 4.1 to 25.7 percentage points by configuration, and focused skills beat large bundles. Test each skill on each model you actually run.

Are agent skills a security risk?

They can be. A January 2026 study by Liu and colleagues analyzed 31,132 skills from two public marketplaces and found 26.1% contained at least one vulnerability, including prompt injection and data exfiltration. Skills with executable scripts were 2.12 times more likely to be vulnerable. Review code, permissions and data access before any skill touches patient data.

Who should write and review clinical AI skills?

Credentialed clinicians matched to the skill's specialty should write or review the clinical rules, with pharmacists for medication logic and scientists for research workflows. A separate reviewer should write the test cases and expected answers, so the author is not grading their own work. Record names, credentials and review dates for every skill.

How often should a clinical skill be re-tested?

Re-test whenever the underlying guideline changes, the model behind the agent changes or anyone edits the skill. Set a fixed review date as well, since guidance can change without anyone noticing. Keep a private test set for each skill and log every run, so you can show what was tested, on which version and by whom.

What did b.well launch with health.skills?

On November 17, 2026, b.well Connected Health launched health.skills, a library of expert-written skills that teach AI assistants to follow protocols and safety checks. Core skills draw on CDC, USPSTF, HEDIS and AHA PREVENT guidance, run against a person's FHIR record and stop when required data is missing. Organizations can also write their own skills.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights