Expert data evaluation imagery representing specialist review of human-generated training data.

Expert data · · 8 min read

Human-generated training data: how to verify your expert data is actually human

Acumen Editorial · Updated November 21, 2026

Teams buy expert data for one reason. They want judgment a model does not already have. A physician's ranking of two answers, a scientist's critique of a protocol, a pharmacist's correction of a dosing error. If that judgment was quietly produced by another model, the dataset teaches your model to agree with itself.

That risk is no longer hypothetical. In September 2026, 404 Media reported that contractors hired to improve ChatGPT were fired for using AI to do the work. Computerworld, republished by CIO, reported that the rules told reviewers not to use AI for reviewing or writing feedback, including Grammarly and AI translation. One contractor told 404 Media that AI use is "the one thing that will get you kicked off ASAP."

If a frontier lab with thousands of contractors catches this inside its own programs, every buyer of expert data should assume the same pressure exists in theirs.

How common is AI use by human annotators?

More common than most buyers assume, and hard to see from the output alone.

The best-known measurement is older but clear. In a 2023 study, Veselovsky, Horta Ribeiro and West at EPFL reran a summarization task on Amazon Mechanical Turk. Using keystroke data and a synthetic-text classifier, they estimated that 33 to 46 percent of crowd workers used large language models to complete it.

Since then, the work has moved up market. Handshake's CEO said in July 2025 that labs now want experts in medicine, law and STEM, with contractor pay averaging $100 to $125 an hour, per Business Insider. Bloomberg Law reported in October 2026 that the expert tier has reached $400 an hour, citing a radiologist among the workers. Higher pay buys real expertise. It also raises the reward for finishing tasks faster than a careful expert would.

Survey research shows where this leads. Pharma insights teams already face AI agents that pass attention checks, as we covered in HCP survey fraud. Training data has the same weakness, with one difference: nobody downstream reads the output as a finished study.

Why does AI-written training data hurt a model?

Because it removes the signal you paid for.

  • Preference data loses its teacher. A reward model learns whoever made the choice. If a model made it, you are training on that model's preferences, errors included.
  • Rationales converge. Model-written explanations tend to sound alike. Real experts disagree, hedge and cite specific clinical context. That spread is useful information.
  • Errors become invisible. A model's mistake, repeated by a second model acting as a "reviewer," looks like agreement. Agreement scores rise while quality falls.
  • Evaluation gets inflated. If a model helped write the gold answers, models from the same family will score well against them.

Computerworld framed this as a version of model collapse, where models trained on AI output degrade. For healthcare and pharma, the practical cost is simpler. You cannot defend a dataset to a regulator, a customer or your own safety team if you cannot say who produced it.

Can AI detectors prove data is human?

No. Treat them as one weak signal at most.

The contractor rules described by Computerworld told reviewers not to use AI detection tools and called them unreliable. Our HCP fraud research found the same pattern in surveys: checks on what a response says are failing, while checks on who produced it and how still hold. A polished clinical rationale is easy to generate. A verified clinician, a plausible work session and a reviewer who can question the answer are hard to fake.

What checks actually verify human-generated training data?

Layer checks on the person, the process and the product. No single layer is enough.

LayerCheckWhat it provesLimit
PersonIdentity and license verification before work startsA real, qualified expert holds the accountDoes not stop that expert from using AI
PersonAccount controls: one person per account, no task resaleThe verified expert is the one workingNeeds monitoring, not just a contract clause
ProcessWritten AI-use policy signed by every contributorExpectations are explicit and enforceableOnly as strong as enforcement
ProcessWork telemetry: time on task, edit history, paste eventsThe work pattern looks like human effortPrivacy rules apply; must be disclosed
ProcessLive or recorded calibration sessionsThe expert can reason aloud on similar itemsCostly, so sample it
ProductBlind expert re-review of a random sampleA second specialist agrees with the judgmentTwo experts can share a blind spot
ProductDisagreement and rationale diversity analysisAnswers vary the way real experts varyLow variance is a flag, not proof
ProductAI text detectorsSome machine-written textUnreliable; never use alone
RecordProvenance metadata on every itemWho, when, which guideline version, which reviewerMust be designed in from the start

The table points to one rule. Prove the person and the process, then sample the product. Detectors on the product alone prove very little.

Does workforce structure matter to buyers?

Yes, because it affects continuity and accountability, not just the vendor's legal exposure.

Bloomberg Law reported in September 2026 on a new crop of lawsuits alleging that AI training firms misclassified data annotators, engineers and other workers as contractors. Bloomberg Law reported a similar proposed class action filed in California in May 2025. These are allegations, and the outcomes are open.

For a buyer, the lesson is practical. Ask how contributors are engaged, how they are paid and who supervises their work. A vendor that cannot describe its workforce clearly will struggle to describe its quality controls clearly. And a program that depends on one pool of experts should know what happens if that pool changes mid-project.

Checklist: what to ask an expert data vendor

Use this in procurement and again at the first delivery.

  • How do you verify identity and credentials before an expert's first task?
  • Do you have a written AI-use policy for contributors, and can we see it?
  • How many contributors were removed for AI use or account sharing in the last year?
  • What work telemetry do you collect, and is it disclosed to contributors?
  • What share of items gets blind re-review by a second specialist?
  • How do you report reviewer agreement, and do you flag suspiciously uniform rationales?
  • Does every record carry contributor ID, timestamp, guideline version and reviewer ID?
  • How are contributors engaged, paid and supervised?
  • If an item is later found to be AI-generated, what do you replace and at whose cost?
  • Can you show all of this on a sample dataset before we sign?

A vendor that answers with "our experts are vetted" has answered half the question. The other half is how the work was done.

How Acumen helps

We deliver verified expert data for healthcare, life sciences and pharma: clinicians, scientists and pharma specialists matched to the task. Every contributor is identity- and credential-verified before they work. Our process combines a written AI-use policy, specialist re-review, agreement reporting and per-record provenance, so you can show who produced each judgment and how it was checked.

For post-training teams, our RLHF data for healthcare service delivers clinician-ranked preferences, written rationales and rubric scores with that provenance attached. For the wider sourcing question, see how to source expert-verified healthcare AI training data.

Request a sample dataset Schedule a conversation

FAQ

What is human-generated training data?

Human-generated training data is data whose labels, rankings, rationales or answers were produced by real people rather than by a model. In expert data, it means a verified clinician, scientist or specialist made each judgment. Buyers pay for it because it carries judgment a model does not already have, so its value depends on proving who did the work.

How common is AI use by data annotators?

It is common enough to plan for. A 2023 EPFL study estimated that 33 to 46 percent of crowd workers on Amazon Mechanical Turk used large language models on a summarization task. In September 2026, 404 Media reported that contractors working on ChatGPT were fired for using AI to do the work.

Can AI detectors tell whether training data is human?

Not reliably. The contractor rules reported by Computerworld in September 2026 described detection tools as unreliable. Detectors can flag some machine-written text, but they miss edited output and can misfire on careful human writing. Use them as one weak signal alongside identity verification, work telemetry, blind expert re-review and per-record provenance.

Why does AI-written training data hurt a model?

It removes the signal you paid for. A reward model trained on model-made preferences learns that model's choices and errors. Model-written rationales converge, so real expert disagreement disappears. And when a model helped write evaluation answers, related models score well against them, which inflates results and hides real weaknesses.

What should we ask an expert data vendor about AI use?

Ask for the written contributor AI-use policy, how it is enforced and how many contributors were removed for AI use or account sharing last year. Ask what work telemetry is collected, what share of items gets blind specialist re-review, and whether every record carries contributor, timestamp, guideline and reviewer IDs. Then ask to see it on a sample.

Does a vendor's workforce model matter to buyers?

Yes. How contributors are engaged, paid and supervised affects quality control, continuity and accountability. Bloomberg Law reported in September 2026 on lawsuits alleging AI training firms misclassified workers as contractors; these are allegations with open outcomes. Buyers should ask how the workforce is structured and what happens if the expert pool changes mid-project.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights