What automation actually means for an OSINT analyst

The Analyst's Stack · Foundations

I get asked some version of this question most weeks, usually by someone who is very good at investigation and has decided they are bad at computers:

How do you actually use automation and AI in this work?

The honest answer is that the AI part is small and comes late. Almost all of the value is in something much more boring, and the boring thing is learnable in an afternoon. This series is the long answer. This post is the shape of it.

First, what it is not

Automation is not a robot doing your job. It is not a model that reads 1,000 documents and tells you which one matters. When people imagine automation and recoil, they are usually imagining replacement, and then quite reasonably concluding that their judgement cannot be replaced. They are right. That is not what this is.

Automation is also not “being technical.” The most useful automated thing I have ever built was 40 lines long and did one thing: it checked a website every 15 minutes and told me when something new appeared. It made no decisions. It had no opinions. It just meant that I found out in 15 minutes instead of the following Tuesday, and that turned out to be the entire difference between a story and a missed story.

Here is a better definition:

Working definition

Automation is moving the act of looking from your attention to a machine’s, so that your attention is spent on the part only you can do — deciding what a thing means.

That is it. Everything else in this series is detail.

Every collection system is the same 6 stages

Once you have built a few of these, you notice something slightly deflating: they are all the same system. The subject changes, the sources change, the language changes, and the shape does not. Every collection system I have built, and every one I have taken apart, is some version of 6 stages in a row. I call that shape the spine, and I will keep calling it that for the rest of the series.

Six stages in a row, left to right: Collect, Normalize, Store, Analyze, Alert, Present. Analyze is highlighted. — **The spine.** Every post in this series refers back to this. Plenty of systems skip a stage. Almost none reorder it, and part 9 says so on its page.

Read left to right:

1 Collect

Get the raw thing. A page, a post, a file, an image, a row from an API (a structured way for one program to ask another for data). This stage is where most of the fighting happens — sites that block you, sources with no export button, formats that change without warning — but conceptually it is the simplest step in the chain.

2 Normalize

Turn the raw thing into one consistent shape. A post from one platform and a post from another arrive looking nothing alike; after this stage they both have a date, an author, a body, a link, and a source. Everything downstream depends on this, and almost everyone skips it at first.

3 Store

Write it down somewhere that survives the program exiting. Usually a small database. The critical part is not the storage — it is giving every item a stable identifier so that you can tell, tomorrow, whether you have already seen it.

4 Analyze

Narrow the pile. Translate, classify, extract names and places, group near-duplicates, score for relevance. This is the only stage where AI genuinely belongs, and even here it is narrowing, never deciding. Two later posts — where AI actually fits in OSINT and the do’s and don’ts for investigators — are about this stage and nothing else.

5 Alert

Tell a human that something crossed a threshold. A message, an email, a row turning red. If nothing ever reaches a person, you have built a very expensive way of generating logs.

6 Present

Make it readable. A map, a table, a timeline, a feed. This is where an investigation actually happens, and it is the stage most often treated as an afterthought.

You will see this diagram again on nearly every post in this series, because the fastest way to understand an unfamiliar tool is to work out which stages it covers and which it leaves to you.

The 2 stages nobody wants to build

Ask someone to describe the system they want and they will describe stages 1, 5 and 6: get the posts, ping me, show me a map. Nobody has ever asked me for stage 2 or 3. They are also the 2 stages that decide whether the thing survives contact with a month of real use.

Here is why, in the most concrete way I can put it.

Say you are watching a site that posts new entries. You write something that fetches the page every 15 minutes and messages you about each entry it finds. You run it. It works. You go to bed.

By morning you have 96 identical messages about the same entry, because “the entries currently on the page” is not the same question as “the entries I have not seen before.” The program has no memory. Every run is its first run.

The fix is stage 3, and it is smaller than you would guess: give each entry an identifier that does not change, write it down, and skip anything you have already written down. Now running the program twice produces nothing the second time. That property has a name — idempotency (running the same job twice leaves you in the same state as running it once) — and once you start looking for it you will notice it is the difference between tools that people keep using and tools that get switched off in week 2.

The cheapest possible version

The identifier does not need to be clever. A hash of the entry’s URL plus its title is usually enough — a hash (a short fixed-length fingerprint of some text; the same input always produces the same fingerprint, and a different input almost never does) gives you a stable key even when the source gives you nothing to work with. Store those keys in a file if you like. It does not have to be a database to be a database.

Two entries from the same page. The first entry's URL and title are joined and hashed into a short key. Running the job again on the same entry produces the same key, so it is skipped. A second, different entry produces a completely different key and is treated as new. — **The same entry always lands on the same key.** Both keys here are real: run SHA-256 over the string on the left and the string in the middle is the first 12 characters of the result.

Stage 2 earns its keep the moment you add a second source. As long as you have one source, normalizing feels like pointless ceremony. Add a second and you either normalize or you write every downstream piece of logic twice. Add a fifth and the version without a normalize stage is no longer repairable.

Naming the stages costs nothing and buys you one thing: a run that tells you where it got to. Here is a toy version of the whole spine — 2 invented feeds that arrive in different shapes, one line printed as each stage finishes.

$ python3 spine.py
collect    2 sources polled, 7 raw items
normalize  7 items in one shape: source, title, link, date, author
store      7 seen, 7 new, 0 already known
analyze    7 new items narrowed to 3 worth a look
alert      3 alerts written to alerts.txt
present    wrote report.html

One run, printing as it goes. The value of naming the stages is that a failure tells you which one broke.

Run it a second time and the store line reads 7 seen, 0 new, 7 already known, and the analyze and alert lines fall to zero. That is idempotency, printing itself.

Where AI actually comes in

Stage 4, and only stage 4.

I will spend 2 whole posts on this later, so here I will just plant the flag. In this work, a model is good at reducing a pile you cannot read into a pile you can. Translate 400 posts so you can skim them. Sort 10,000 items into “probably relevant” and “probably not.” Notice that these 6 records are describing the same event under 6 spellings.

It is not good at telling you whether something is true. It can attempt verification and attribution, but what comes back is a lead, not a finding, and checking it is your job. The moment a model’s output becomes evidence without a human between it and the conclusion, you have automated the one part of the job that was never supposed to be automated.

AI narrows the pile. A person decides what is in it. If you cannot point at the human who checked a given claim, the claim is not checked, and “the model was confident” is not a sourcing note you can publish.

What the spine does not tell you

It is a shape, not a plan, and it is silent on the 3 things most likely to sink the project.

It says nothing about whether you are allowed to. Six stages will happily collect something you should not be holding. The legal and ethical question is not a stage; it sits before stage 1 and the diagram cannot remind you of it.

It says nothing about cost. A pipeline with all 6 stages can run for pennies or can quietly spend more on model calls in a weekend than the story is worth. Nothing in the shape tells you which.

A complete system can still be useless. I have built things with all 6 stages working perfectly that nobody opened twice, because the question they answered was one I had invented. The spine tells you whether a tool is finished. It cannot tell you whether it was worth building.

When not to automate

This is the part that gets left out of tutorials, because tutorials are written by people who want you to finish the tutorial.

Two rows of conditions. Automate when it repeats, the source holds its shape, the volume is past eyeballing, and you would notice it stopping. Do not automate when the question is asked once, the volume is a handful, the source changes shape weekly, or the legal ground is unsettled. — **Four reasons to put the laptop down.** Any one of the bottom row is enough.

The question gets asked once. Automation is an investment that pays back over repetitions. If you need this answer today and never again, open the site and read it. I have watched people — I have been people — spend 6 hours building something to avoid 90 minutes of clicking.

The volume is small. Forty items is not a data problem. Forty items is an afternoon. The threshold where a machine beats a careful human is higher than it feels when you are bored.

The source changes shape every week. Some sites are stable for years. Others rewrite their front end constantly, and every rewrite breaks your collection silently — which is worse than breaking loudly, because a tool that quietly returns nothing looks exactly like a week with no news. If you cannot detect that failure, do not build on that source.

The legal or ethical ground is unsettled. Terms of service, the rules that apply to personal data where you work, and the question of whether collecting at scale changes what a thing is — all of that gets decided before the first line of code, not after someone asks you about it. This is not a technical constraint and you cannot engineer around it.

Of the 4, 2 are about restraint and the other 2 about honesty, which is roughly the ratio I would expect.

What is in the rest of this series

There are 22 posts, of 2 kinds that work differently.

The guide is 10 teaching posts. Four more foundations follow this one: building a first workflow end to end, where AI fits, the do’s and don’ts, and the OPSEC problem that automation creates (you have automated the act of looking, so you have automated the act of being seen). Then 5 methods that go deeper — automating without writing code, running models on your own machine, reaching sources that block you, measuring whether an AI step actually works, and storing what you collect so you can find it again.

The showcase is 12 short posts, one per working system. Why it was built, what it solves, how it fits together, and what it cannot do.

If you read only two, read the next one and the one on measuring AI steps. The first gets something running. The second is the one that will stop you publishing something wrong.

Start here

If you want something concrete before the next post lands: pick one source you check by hand more than twice a week. Not your most important one. Your most repetitive one. Write down, in plain words, what you do each time you check it, one line per step.

You have just written most of the spine. The rest is typing.

If you have never written Python and want to run the code in the next post, it lists 3 free courses in the order I would take them: futurecoder from a standing start, PyBasic for practice, then Python for OSINT in 21 days for the same skills aimed at investigation work.