Where AI actually fits in OSINT

The first post in this series laid out 6 stages. The second built 4 of them in about 60 lines of code. Neither one used AI anywhere, and that was not an oversight.

This is the post where it arrives. It arrives in exactly one place, nothing it hands back is a finished answer, and the boundary around it is sharper than most of what you will read elsewhere.

One stage out of 6

Stage 4. Analyze. Narrowing the pile.

It is worth being specific about why it is not any of the other 5.

Collection is a network problem. A model does not fetch a page on its own. Ask one to read a URL it has no way of reaching and it will describe the page it expects to be there, fluently, in the right format, wrong. The failure is invisible because the output looks exactly like success.

Normalization is a shape problem with a correct answer. Pulling a date out of one site’s HTML is a job for a line of code that either works or crashes. A model does it too, more expensively, and fails in a way you cannot see from the output.

Storage is a database. Alerting is a threshold and a webhook. Presentation is a table and a map. None of these are judgement calls, and all of them are cheaper, faster and more inspectable without a model in the loop.

That leaves stage 4. Inside stage 4 there is a specific list of jobs, and there is a test that puts each job on the list or off it.

The test

Here is the whole thing:

I want to make the “roughly right” part concrete, because it is doing all the work.

Suppose a step is wrong 1 time in 10. (That is an illustrative figure, not a measurement — measuring your own step is a later post in this series and the number you get will surprise you.) Ask what a single wrong answer actually costs.

If the step translated a post and got a clause wrong, the cost is that you read a slightly garbled sentence and go find someone who reads the language. Annoying. Recoverable.

If the step decided a post was irrelevant and dropped it from your queue, the cost is that you never saw it. Not recoverable — and worse, invisible, because a pile you never saw looks identical to a pile that was never there.

Same error rate. Completely different consequence. Which is why the list below is not “things models are good at” but “things where a model being wrong is a nuisance instead of a hole in your reporting.”

What fits

Translation

The clearest case. You have 400 posts in a language you do not read, and your options are: skip them, wait 3 days for a human translator, or machine-translate all 400 in a minute and skim them.

The machine version is worse than the human version. It is also available now and at volume, and skimming 400 rough translations to find the 6 worth a real translation is a genuinely different job from translating 400 posts. You are not publishing the machine output. You are using it to decide what to hand a person.

The rule that keeps this safe: never quote a machine translation. The moment a sentence is going into your copy, it gets translated by someone who reads the language and will put their name next to it.

Transcription

The same shape, with audio and video. An hour of speech becomes searchable text in a few minutes, you find the 90 seconds that matter, and then you listen to those 90 seconds yourself.

Transcription has one failure mode worth knowing about: it is most wrong on exactly the words you care about. Names, places, technical terms, numbers — anything rare. The connective tissue comes out fine, and the proper noun that is the reason you are listening comes out as something that sounds like it. Search the transcript, then trust your ears.

Entity extraction

Pulling the names, places, organizations, dates and identifiers out of unstructured text, so you can filter and count and cross-reference instead of reading.

This one fits the test almost perfectly, because an extraction error is a shortlist error. A missed name means you filter on it and see nothing, which you will notice the first time you read one of the documents by hand. An invented name means an extra row that does not survive being checked against the source. Both are survivable, as long as the extracted entity is a pointer back to the document and not a replacement for it.

Keep the offset, the sentence, or at minimum the source document ID next to every extracted entity. An entity list with no way back to where each entity came from is a set of assertions, and you have just built a machine that generates assertions.

Triage classification

Sorting a large queue into probably relevant and probably not, so a person reads the first pile first.

This is the highest-value AI step in most real systems, and the most dangerous one, because it is the step where being wrong means you never see something. Two rules make it safe:

Deduplication

You are watching 12 sources. Six of them report the same event. You want one row, not 6, and the 6 do not share a headline, a spelling, or a timestamp.

Exact matching fails here — that is what stage 3’s stable identifier is for, and it only catches genuinely identical items. Near-duplicate grouping needs something that understands that “a fire at the depot” and “blaze reported at storage facility” are the same event, which is where embeddings come in.

An embedding (a way of turning a piece of text into a list of numbers, arranged so that two texts meaning the same thing end up with numbers close together) lets you ask a question you cannot ask with string matching: which of these items are near each other? Group the near ones, show the group as a single row, keep every original underneath it.

It fits the test because the failure modes are both visible. Two events wrongly merged produce one row that reads strangely when you open it, and you split them. One event wrongly split produces 2 rows that a human notices as duplicates within about a second. Neither one loses you anything, provided the grouping is a display layer over the originals rather than a destructive merge.

Summaries, for triage and nothing else

A summary is a reasonable way to decide whether to read something. It is not a reasonable way to have read something.

The distinction sounds pedantic until you watch it fail. A summary is a compression, and compression is lossy in a direction you did not choose. The hedge that made the claim survivable, the attribution that made it sourced, the date that made it relevant — those are exactly the kind of detail that a summary drops, because a summary optimises for the gist and the gist is not what you are in this job for.

So: summarise to sort. Read the original before anything based on it leaves your desk. If you find yourself writing from the summary, the summary has quietly become the source and you no longer have one.

Vision, for coarse screening

Machine vision earns its place on very blunt questions asked of very large image sets. Does this frame contain a vehicle. Is this a screenshot or a photograph. Does this image have text in it. Is this the same scene as that one.

Coarse is the operative word. “There is a building here” is a screening question that saves you an afternoon of scrolling. “This is that building” is an identification, which is a claim, which puts it on the other list.

What a model can draft but not sign off

Three jobs sit on the other side of the line, and one rule covers all of them.

Be clear about what the objection is, because it is not that a model cannot do these. A current frontier model will attempt all 3, and it sometimes does them well, better than a tired analyst’s first pass. The objection is to what comes back. It is a lead, not a finding, and when it is wrong nothing in the text says so. Before it becomes evidence it needs the same review you would give a tip from a stranger, and that review is most of the work.

Verification. Ask a model whether something is true and it can do real work: find where a claim seems to have started, compare dates and details, flag that a photo may predate the event it supposedly shows. Some of those answers are right. You cannot tell which by reading them, because the wrong ones come in the same voice as the right ones. So the answer is a list of things to check. Open every source it cites yourself. Confirm the source exists and says what the model says it says, checking against the original rather than the model’s paraphrase of it. And write down what you opened and when. A model that agrees with your hypothesis is not corroboration, it is an echo, and it is a more convincing echo every year.

Attribution. Who did this, who runs that account, which group is behind this infrastructure. Attribution is a chain of specific, checkable links, and the entire value of it is that each link can be attacked and survive. A model can assemble a candidate chain from the material you give it, and some of those chains hold. It reads identically whether the links hold or not. Reviewing it means taking the links one at a time and reproducing each one yourself: rerun the lookup, pull the registration record, find the post again, and note how you got there. Anything you cannot reproduce comes out. Do that and what survives is your attribution, with a head start. Skip it and you have published the shape of one.

Editorial judgement. Whether something is worth publishing depends on your audience, your standard of proof, and what it costs to be wrong. A model can lay out the arguments on both sides. It holds none of those 3 things and will answer anyway.

Anything that becomes evidence without a human in the chain. This is not a fourth job. It is the rule that covers the 3 above. The question was never what the model can do. It is whether you can name the person who checked a given claim. If you cannot, the claim is not checked, however good the model that produced it.

Count that checking in the cost. Reviewing a model’s verification properly can take as long as doing it yourself, and sometimes longer, because you are checking against an answer that has already told you what to expect. Sometimes it is still worth it: the model found a source you would have missed, or cleared 3 dead ends in a minute. Sometimes it is not. That is the real decision on this list. Not whether the model is capable, but whether its draft plus your review beats your review alone.

Why the second list still needs a person

Everything on the first list produces a shortlist. Everything on the second produces a claim.

A shortlist is a suggestion about where to spend attention. It can be wrong and the cost is some wasted time. Nobody reads a shortlist and believes something because of it.

A claim is a statement about the world, and the only thing that makes it usable is that somebody is standing behind it. When a claim comes out of a model with nobody between it and your report, there is no one standing behind it. You cannot publish it, you cannot defend it when it is challenged, and you cannot tell an editor or a court or a reader where it came from, because the honest answer is a model said so, and that is not a source.

This is also the practical reason the distinction survives contact with a deadline. It is not an ethical preference that a better model will dissolve. A better model writes a better draft; it does not turn the draft into a source. That takes a person who checked it.

Building the gate so it is real

“A human reviews it” is easy to say and easy to let rot. Three properties make a review gate something other than a rubber stamp.

The human sees the original, not the model’s version of it. If your interface shows a summary with a link to the source, the summary is what gets read. Show the source and put the model’s output beside it.

Disagreement is recorded. Every time a person overrules the model, that gets written down as a row. Two reasons: it is the only way you will ever find out how good the step actually is, and a queue of disagreements is the most interesting thing in the system.

The gate can be slow. The moment the volume forces people to approve in bulk, you no longer have review — you have a person clicking a button that a model already pressed. If throughput is the constraint, the correct move is to narrow harder upstream so fewer items reach the gate, never to speed the gate up.

How to add the first one

If you have the 60-line monitor from the last post running, the smallest useful AI step you can bolt onto it is a translation call between “store” and “notify” — the item is already stored, the translation goes in an extra column, and the alert carries both. Nothing is destroyed if the translation is bad, the original is one click away, and you will learn more about where models fail from a fortnight of that than from any amount of reading.

The step after that is a relevance score, and it is a bigger jump than it looks, because it is the first one that can lose you something. Do not build it until you have somewhere to put the items it scores low.

The next post is about the failure modes: hallucination as something you design around rather than complain about, prompt injection arriving through the pages you scrape, keeping the evidence chain intact, and what this costs per month.