
The Analyst's Stack Part 21 of 22
Does your AI step actually work?
You added a step to your pipeline. It runs, it produces output, and the output looks right. You skim 10 items, 9 of them are fine, and you move on.
That is not a measurement. It is an impression, and you would have formed exactly the same impression if the step were wrong a third of the time, because the 10 items you skimmed were the 10 that were easiest to skim.
This post is about turning that impression into a number, using arithmetic you already know. It ends with the time I did it to one of my own steps and got an answer I did not enjoy.
The 50 items you labelled by hand
Everything here rests on one unglamorous object: a gold set (a small batch of items where you have written down, by hand, what the correct answer is). It is the only thing in your system that is not an opinion.
Fifty items is enough to start. Not 5,000, not the whole corpus. Fifty.
Build it like this. Pull a sample of items your step will actually see — not a curated sample, a boring one, taken across different days and different sources, including the awkward ones you would normally skip. Then go through them yourself and write down the right answer for each. Is this item relevant, yes or no. Is this extracted name really a name. Is this really the same event as that one.
Here is the start of one, built from invented items: made-up headlines about a fictional town council, a fictional shipping company and a fictional storm. The rule it was labelled by is written down first, as the steps below ask: relevant means the item reports a disruption to port or shipping operations that has already happened.
| # | item | relevant | note |
|---|---|---|---|
| 1 | Marrowby Town Council approves new bus timetable for the harbour ward | no | transport but not the port |
| 2 | Greyling Shipping suspends Marrowby ferry crossings after engine fault | yes | |
| 3 | Storm Varrow expected to reach the coast on Thursday night | no | forecast only; nothing disrupted yet |
| 4 | Harbour master closes Marrowby outer berths as Storm Varrow arrives | yes | |
| 5 | Council meeting on quay resurfacing postponed for a second time | no | a vote delayed is not a disruption |
| 6 | Greyling Shipping names new chief executive | no | |
| 7 | Two cargo sailings from Marrowby cancelled as gale warning is upgraded | yes | |
| 8 | Marrowby library extends weekend opening hours | no | |
| 9 | Greyling Shipping reports higher summer passenger numbers | no | |
| 10 | Storm Varrow: fallen trees block the coast road north of Marrowby | no | road not port; hard case |
| 11 | Ferry passengers stranded overnight after Greyling cancels last crossing | yes | |
| 12 | Town council publishes draft budget for next year | no | |
| 13 | Marrowby harbour reopens after Storm Varrow damage checks | yes | a reopening still reports the closure; hard case |
| 14 | Greyling Shipping orders two new freight vessels | no | |
| 15 | Council votes to rename the harbour car park | no |
Three rules make the difference between a gold set and a pile of rows.
It is boring, it takes an afternoon, and it is the only thing that converts “it looks like it works” into a sentence you can defend to an editor.
Two numbers, in plain words
Once you have the labels, every item your step handled falls into one of 4 boxes. There is standard jargon for these and I am going to use plain descriptions instead, because the jargon is where people lose the thread.
- Correctly flagged. The step said yes and it was yes.
- Wrongly flagged. The step said yes and it was no.
- Correctly ignored. The step said no and it was no.
- Wrongly ignored. The step said no and it was yes.
From those 4 counts come the only 2 numbers you need:
The 2 numbers
Precision — of what it flagged, how much was right.
Recall — of what was there, how much did it catch.
Precision asks how much of the pile the step handed you is rubbish. Recall asks how much of the pile that existed never reached you. One is about wasted attention. The other is about things you never saw.
A worked example
Invented numbers, to show the arithmetic. These are an example, not a measurement of anything.
You have your 50 items. You labelled 12 of them relevant. You run the step, and it flags 15. You line the 2 columns up and count:
| Correctly flagged | 9 |
| Wrongly flagged | 6 |
| Wrongly ignored | 3 |
| Correctly ignored | 32 |
Precision: of the 15 it flagged, 9 were right. That is 9 ÷ 15 = 0.60: 60 per cent of what it handed you was worth your time.
Recall: of the 12 that were really there, it caught 9. That is 9 ÷ 12 = 0.75. It missed a quarter of them.
Now look at what a single overall accuracy figure would have told you. The step got 9 + 32 = 41 of 50 items right, which is 82 per cent, and 82 per cent sounds like a working tool. It is not. Three relevant items never reached you and you have no idea what was in them.
Push it further. A step that flags nothing at all gets 38 of 50 right — 76 per cent — while being completely useless. That is why the 2 numbers are worth the effort and a single accuracy score is not: accuracy is dominated by the boring majority, and in this work the boring majority is never what you are asking about.
The same example again, out of a script. By this point the gold set has one more column than the table above: beside the item, your label and the note, it records what the step said about each of the 50 invented items. The script lines those two columns up and counts.
$ python3 score.py gold-set.csv
gold set: 50 items, 12 labelled relevant
step: flagged 15
correctly flagged 9
wrongly flagged 6
wrongly ignored 3
correctly ignored 32
of what it flagged, how much was right 0.60
of what was there, how much did it catch 0.75
You have to choose which one to protect
Here is the part that people expect to be a bug and that is actually a decision.
You cannot maximise both numbers. Your step has a cut-off somewhere — a score, a threshold, a confidence — and moving it trades one number for the other. Loosen the cut-off and you catch more of what was there and also drag in more rubbish: recall up, precision down. Tighten it and the pile gets cleaner while more real items fall off the end: precision up, recall down.
There is no setting that fixes this, so stop looking for one. The question is not which number is higher, it is which mistake costs you more, and that is a question about your work rather than about your code.
Most writing about this treats the 2 numbers as interchangeable and blends them into one figure, because in most settings the costs really are roughly symmetrical. In investigative work they are not, and the asymmetry runs in a specific direction.
A wrong output is a claim; a missing one is not
This is the line to keep.
If your step puts a term in a bucket called names, a reader takes it to mean a person.
If it puts an item in a queue called relevant, a colleague takes it to mean somebody
thought it was relevant. The label is a statement about the world, and a wrong one is a
published error with your name near it.
A missing item is different. It is a gap. Gaps are bad, and gaps are what the rest of your method exists to close — the sampling discipline from where AI fits in this work, and the rule that a low-relevance pile stays on screen rather than being deleted. A gap is recoverable by someone doing the work. A false label is already out of the building.
So when the 2 numbers pull against each other, prefer losing recall. Tighten the cut-off, accept that you will miss more, and keep the bucket honest. Then design the rest of the system on the assumption that the bucket is incomplete, because it is.
The reason this is not the default advice
The time I measured my own step and did not like the answer
Now the part I would rather not write.
The multilingual monitoring
system elsewhere in this series has
a step that sorts terms out of collected articles into buckets — names, locations, a
few others — so that someone scanning a day’s collection can see at a glance who and
where an article is about. It shipped. It populated in all 3 languages.
Spot-checking it looked fine.
Then I ran it across the real corpus, 3,900 articles, and counted properly against hand-labelled answers.
Roughly a third to a half of what it had put in the names bucket was not a name.
Not near-misses. Function words. In the English portion: WHO, WHILE, TOLD. The
other 2 languages had their own equivalents — the word for “the government”, the word
for “house” — sitting in a bucket that a reader would take to mean people.
The obvious fix made it worse
The reflex is to raise the confidence threshold: keep the high-scoring candidates, drop the tail, move on.
I tried it and the bucket got worse, and the measurement said exactly why. The step scored candidates partly by counting designators (words that tend to sit next to a person’s name — a title, a role, a reporting verb) in the surrounding text. Two rows from that run, with the real name withheld here:
told cap=0 low=7 designated=3 <- classified as a name
[a real person's name] cap=0 low=0 designated=2 <- an actual name, scored LOWER
A function word outscored an actual person on the signal the threshold acts on. So raising the cut-off removed real names first and kept the noise. Every turn of the dial made the bucket less honest while making the score look more selective.
The generalisable bit
The replacement needed 3 separate signals instead of one: capitalisation evidence where the writing system has capitals at all, the designator count divided by total occurrences where it does not, and the definite article to tell a title attached to a following word from a title sitting beside a name. A single number could not carry it, because a single number was what had gone wrong.
That step was rules, not a language model
I want to be exact about this, because it would be easy to let you assume otherwise and the assumption would flatter me in the wrong direction.
The step was rule-based. Designator counts, capitalisation checks, article handling. No model produced those errors, and I am not offering you a story about a model behaving badly.
The example is stronger for it, for 3 reasons.
The measurement is identical either way. A gold set, 4 counts, 2 numbers. Nothing in this post changes when the thing being measured is a prompt instead of a rule.
The failure shape is the shape a model step fails in. One confident score, applied to every item, ranking a wrong answer above a right one, with a threshold sitting on top of it that looks like a safety control and is not. If you have a classifier returning a relevance score between 0 and 1, that is the machine you are running, and its ranking can be upside down in exactly the same way.
And nobody gets to wave this one away. “That model was bad, the new one is better” is available for every model failure ever published, and it is the reason those stories teach so little. This was code I wrote, whose logic I could read line by line, and it was still wrong a third to a half of the time in one bucket for months, because I had looked at it instead of counting it.
If anything, rules were the easy case. The step printed designated=3 next to its
answer, which is how I could see what was happening within an hour. A model would have
returned a confident label and a number between 0 and 1, and I would have had the
same broken ranking with nothing to read.
A score has a date on it
One more thing, and it is the one that gets skipped.
Your number is not a property of the step. It is a property of the step, measured against that set of items, on that day. Sources rewrite their pages. Coverage moves to different kinds of stories. A step that measured well in March can be quietly wrong by September without anyone touching a line of its code.
So put it in the calendar. Every few months, and after any change to the step, the prompt, the model version or the sources it reads, do the same 50 items again. Add 20 fresh ones sampled from recent collection so the set does not become a museum of last year’s material. Write down the date next to the figures.
The history is the useful artifact. A single precision figure tells you very little; the same figure holding steady across 3 measurements and then dropping on the fourth tells you something happened, and roughly when.
What measuring does not give you
Fifty items is a small sample, and a small sample gives you a rough figure, not a precise one. A measured precision like the 0.60 in the worked example above means “somewhere around 60 per cent”, and the honest way to use it is to compare it against the same set measured again later, never to quote it to 2 decimal places as though it were a specification.
It also only measures what you labelled. If your step is wrong in a way you did not think to check — a whole category of item that never appeared in your sample, a language you do not read — your gold set is silent about it and your score will look healthy anyway. That is not a flaw in the method so much as the reason the sampling has to stay boring and unfiltered.
And the labels are yours. You are one person with a view of what counts as relevant, and a colleague labelling the same 50 items will disagree with you on some of them. Where a decision actually matters, have someone else label a subset and look at where the two of you differ. The disagreements are usually more informative than the score.
None of this makes the step better. It makes the step’s wrongness a known quantity, which is a different thing and the only one on offer.
Where to start, this week
Take the step in your pipeline that would embarrass you most if it were wrong. Not the one you are most curious about. The one whose output somebody downstream reads as a fact.
Sample 50 items it has handled. Label them yourself, before you look at what it said. Count the 4 boxes. Work out the 2 numbers.
There is a fair chance the answer is worse than you expect, and that is the point of doing it. Every number in this series is either real or labelled as an example, and this is what that rule costs: I ran this on my own work and a third to a half of one bucket was wrong. The alternative was not a better step. The alternative was not knowing.
Everything in the do’s and don’ts — traceable claims, a review gate that gets read, pinned model versions — assumes you know roughly how often the step under it is wrong. This is where that number comes from.