How Do You Grade the Graders?
Somewhere in Mozambique, a woman calls a health hotline. Her son has a fever, and she’s heard something on the radio about an outbreak in her district. Is it true? The call takes four minutes. When it ends, a recording lands in an archive, one among thousands.
Locked inside that recording is exactly what a public-health program needs to know: what people are worried about, which symptoms are appearing where, what rumours are spreading faster than the facts. Multiply by thousands of calls and you have a map of a country’s health anxieties, updating daily.
But it’s locked in audio. And not just any audio: Mozambican Portuguese, with callers sliding into Changana or Makhuwa mid-sentence. Speech recognition models, trained mostly on English and a handful of the world’s most widely resourced languages, produce a rough draft at best. To turn these calls into data, a human has to listen, write down what was said, translate it, and pull out the facts:
There’s a name for this work, and a mostly invisible profession behind it: annotation. Every AI system you’ve ever used learned from data that people labeled by hand: millions of images sorted by crowd workers, voice-assistant recordings transcribed in office parks, chatbot answers ranked one by one. The models get the headlines. The annotators make the models possible.
We hired three of them. And that’s when we met the question this post is actually about.
The answer-key problem
Think about how anything gets graded. A teacher marks an exam against an answer key. A driving examiner knows what a correct parallel park looks like. Grading is easy when someone already knows the right answer.
Annotation breaks that. We hired annotators precisely because we don’t have the answers. Their transcripts are the answer key, the ground truth our models will be trained and judged against. If an annotator mishears “measles” as “malaria”, that error doesn’t get caught downstream. It becomes the truth downstream.
So: how do you grade someone whose job is producing the answers?
When we started, we weren’t even set up to ask the question. Our internal tool had one annotation per audio, anyone logged in could edit anything, and the last person to hit save won. There were no roles, no history of who changed what, and no quality signal at all. That was fine for three trusted teammates, and not fine once you’re paying people per batch of work.
First idea: make them agree
The textbook answer is inter-annotator agreement: give the same audio to all three annotators, compare their answers, and trust the data when they agree. There’s half a century of statistics for this (Cohen’s kappa, Krippendorff’s alpha) and a lovely survey by Artstein & Poesio (linked below) that I spent a few evenings with.
The more I read, the less it fit:
| Peer agreement (kappa / alpha) | Hidden answer keys (what we built) | |
|---|---|---|
| Measures | do annotators agree? | are they right? |
| With 3 annotators | shared mistakes look like consensus | mistakes show against the key |
| Free text (transcripts, translations) | awkward; the math wants categories | reference metrics exist (CER, COMET) |
| Cost | every agreement item is done 3× | the answers already exist |
The first row is the killer. Agreement measures reliability, not correctness. Three people who all mishear the same muffled word agree perfectly, and are perfectly wrong. With thirty annotators, statistics can rescue you. With three, a shared mistake is invisible.
It turns out the data-labeling industry reached the same conclusion years ago. Appen hides test questions in workers’ queues; Scale runs benchmark tasks; researchers call them honeypots. The trick is the same everywhere: mix items with known answers into the real work, don’t tell anyone which is which, and score people on those. Think of the mystery shopper. The shop assistant serves every customer well because any of them might be the inspector.
We borrowed it wholesale.
Hiding the answer keys
Here’s what an annotator experiences: claim a batch of 30 audios, work through them in the browser, claim the next batch when done (two per day, tops). What they don’t experience is any difference between the audios. But there is one:
The 3 gold audios were already annotated by trusted people: doctors, admins, our most senior annotator. The new annotator does them blind, like any other call. Afterwards, the system compares their answer to the trusted one and scores it.
Two design choices here mattered more than we’d guessed going in:
Gold is derived, not curated. We never maintain a list of “the gold audios”. A call counts as gold because trusted people authored all three of its annotations. As doctors annotate more calls, the pool of answer keys grows by itself, with no curation meetings and no stale lists.
Gold is pinned to a snapshot. The moment a gold call enters someone’s batch, we freeze a copy of the answer, and scoring compares against that frozen copy forever after. The reason is that trusted people keep editing their work, and a doctor fixing a typo next month shouldn’t silently change a score we issued last week.
Scoring: three tasks, three yardsticks
“Compare to the answer key” sounds simple until you try it. What does it mean for a translation to match? “The caller asks about malaria symptoms” and “the person wants to know the signs of malaria” share almost no words, and both are right. One number for “annotation quality” would be a lie. Each task fails differently, so each gets its own measure:
| Task | Metric | In plain terms | Bar |
|---|---|---|---|
| Transcription | 1 − character error rate | how many keystrokes to fix it | 85% |
| Translation | COMET (a neural evaluation model) | does it mean the same thing? | 90% |
| Extraction | exact match per field, F1 for lists | found 2 of 3 symptoms + 1 extra = partial credit, precisely counted | 90% |
(Character-level rather than word-level for transcripts, because in code-switched speech with no standard spelling, word-level scoring punishes harmless spelling variation absurdly hard.)
The three scores average into one alignment % per audio, which rolls up into a running average per annotator.
About those bars: 85 and 90 are educated guesses, not laws of nature. Krippendorff’s rule of thumb for reliable data is 0.80; industry test-question systems typically demand 80–90%. What we actually engineered is the machinery around the numbers: thresholds are live admin settings, every change is audited, and each score is stamped with the thresholds active when it was issued, so raising the bar next month never retroactively fails last month’s work. A grading system that rewrites the past won’t be trusted by the people it grades.
One more fairness rule: if our translation-scoring service is down, the score is “ungradable”, never zero. Don’t punish annotators for your infrastructure.
And the other 27?
Gold calls grade the person. But on the 27 unique audios, the annotator’s answer is the only one in existence, and someone still has to decide it’s good enough to become the record. We eventually realized “verify” was two features wearing one word:
Note the loop on the right. Besides accepting, a reviewer’s other option is not to reject the work but to send it back, the way a teacher returns a draft with comments in the margin. The feedback lands on the annotator’s copy of the work: an amber “Reviewed by Admin” flag appears on their list, the reviewer’s comment shows up in the recording’s notes, and the item stays theirs to fix. Once they’ve reworked it, they mark it resolved and it returns to the reviewer’s queue, going around again until it’s accepted. The reviewer never silently edits the annotator’s work; every improvement goes back through the person who made the mistake, which is also how they stop making it.
Every recording now wears a badge (Gold, Verified, or Pending) so anyone using the data downstream picks exactly the trust level they need.
The part that generalizes
Nothing above is really about Mozambique, or health, or audio. If humans label your data, any data, the shape is the same:
- Hidden answer keys beat peer agreement when annotators are few and the work is hard. Agreement tells you people are consistent; gold tells you they’re right.
- Match the metric to the task. Text, meaning, and structured fields fail in different ways.
- Freeze your answer keys, or every edit to reference data silently rewrites old scores.
- Make thresholds adjustable but never retroactive. You won’t pick the right bar on day one; change it fairly.
- Give annotators honest outs: “no data”, “wrong language”, “I’m unsure”. Every escape hatch you don’t build becomes a fabricated annotation you can’t detect.
- Send flawed work back, don’t fix it quietly. Feedback that routes through the annotator improves the annotator; corrections made behind their back only improve one datum.
The woman’s call now gets transcribed by someone we can vouch for. Not because we trust them personally, but because we can show the score, the frozen answer key it was graded against, and the reviewer who signed off. Her question about the outbreak becomes one reliable data point on that daily-updating map. And that is all annotation ever is: someone listening carefully, and someone else making sure the listening holds up.
References
Agreement theory:
- Artstein & Poesio (2008), Inter-Coder Agreement for Computational Linguistics: the standard survey.
- Cohen (1960), A Coefficient of Agreement for Nominal Scales (explainer).
- Krippendorff’s alpha: source of the ≥ 0.80 guidance.
Metrics we shipped:
- Rei et al. (2020), COMET: A Neural Framework for MT Evaluation (GitHub).
- Is WER a Good Metric for Speech Recognition in Indic Languages?: the case for character-level error rates.
- Tjong Kim Sang & De Meulder (2003), CoNLL-2003 shared task: canonical F1 evaluation for extraction.
Industry gold-task practice:
- Appen: Test Question Best Practices.
- Scale AI: Data Labeling Guide.
- Google PAIR: Data Collection + Evaluation.
Low-resource speech: