Entity resolution across 18 sources

The Analyst's Stack · Showcase

Eighteen catalogues report the same incidents. Deciding which reports are the same incident is the entire product.

Why I built it

The question is boring and it comes up constantly: has this organisation, or one of its suppliers, turned up in a ransomware claim or a breach disclosure?

There is no shortage of places to look. Public and subscription catalogues index leak-site posts, breach notifications and statutory filings, and they overlap heavily because most of them watch each other. So you check 5, get 5 overlapping lists, and land on the real problem: one incident appears on all 5 under 3 spellings of the company name, 2 different dates, and — twice — a name with half its letters replaced by asterisks.

By hand that means deciding, row by row, whether Acme Industries, Inc. and ACME INDUSTRIES are one company or two. Fine for 40 rows. The failure at scale is silent: you report 5 incidents where there was one, and nobody catches it.

What it solves

From 18 sources, 340,920 observations collapsed into 120,063 canonical events across 90,565 victims.

Those are measured figures from one dated rebuild, not a standing claim — every rebuild moves them, so the system re-measures at request time and stamps the answer with when it was taken. The ratio is the thing. Nearly 3 raw records per canonical event, and every count you take off the raw corpus is inflated by an amount you cannot see.

After resolution you can ask the question once: one row per incident, with every original record underneath it.

One incident, 15 records of it. The collapse is only trustworthy if you can open it back up. Two spellings of the victim and 4 of the group, from 9 sources, resolved into one event. Observed is when the tool collected each record, so it spans 8 September to 7 October 2026; the date each source gave sits in a row detail not shown here. Real data from my own aggregator. The source names are hidden, along with each row’s licence line and the panels between the header and the table.

How it works

The spine again, with the weight in stage 4. Same 6 stages as the foundation post — this one just has 18 inputs.

Collect. One adapter per catalogue, each making exactly one call and returning raw bytes; nothing outside the adapter knows that source’s shape. One hard boundary underneath all of it: nothing is ever collected from threat-actor infrastructure. No leak sites, no Tor. Every source is an index that already did that collection, and a test enforces that rather than a convention.

Normalize. One place in the codebase, and only one. Every record gets a collapsed join key for the victim name, a guarded registrable domain, and a flag saying whether the name arrived masked.

Store. Observations are append-only, one row per record per source, so nothing is overwritten by a later opinion.

Analyze. Two passes. The victim registry decides which records are the same organisation; the clusterer then decides which of that organisation’s records are the same incident. Everything downstream reads the result.

Three bars drawn to scale against each other. The first and longest is 340,920 observations, the raw records from 18 catalogues. The second, a little over a third as long, is 120,063 events after resolution, which works out at nearly 3 raw records per event. The third, shorter again, is 90,565 victims, the distinct organisations behind those events, about 1.3 events each. — **Two ratios, and only one of them is duplication.** Three records per event is the same incident reported over and over. The 1.3 events per organisation is a different thing: partly organisations hit twice, partly merges the system refused to make.

Alert. Watchlist matching on names and domains, deduplicated by content so one incident does not arrive 3 times, then a message to a chat channel.

Present. A dashboard with search, filters and 4 export formats, plus an API.

How I built it

TypeScript and Postgres, small packages that each own one stage. It runs as 3 batch commands — collect, resolve, alert — not as a service, so the interesting failure mode is “the run was wrong” rather than “the daemon is down”.

The 3-year arc matters more than the stack. The 2023 original was a Flask app: a Tor scraper reading leak sites directly, a Postgres table, a dashboard with charts. It worked, and I would not build it that way again, for 2 reasons unrelated to Python.

The first is the boundary. Reading actor infrastructure directly puts you on their timetable and in their logs — and everything that tool went and got, a catalogue had already got. The 2023 tool’s own archive is now 2 of the 18 sources, imported as history, and that is the only form it survives in.

The second is that the old one stored what it had parsed and threw away what it fetched, which made every parser bug permanent: fixing it meant going back for the page, and the page was often gone. So the rebuild keeps the raw bytes — gzipped, content-addressed, named by the hash of what arrived. A parser fix is replayed against stored payloads with no network call at all.

What was hard was picking constants. How many days apart can 2 reports of the same victim be and still be one incident? There is no valley in the data to put a threshold in — the gaps decay smoothly — so any number is a policy choice wearing a discovery’s clothes. The system errs one way, always: prefer a false split over a false merge. A split is visible and correctable. A merge fuses 2 organisations into one row and corrupts the count silently.

$ bun run resolve
$ bun run packages/resolve/src/cli.ts
{
  "observationsTotal": 33,
  "observationsPlaced": 33,
  "eventsTotal": 24,
  "victimsTotal": 20,
  "skipsTotal": 0,
  "skipsByReason": {},
  "reductionPct": 27.27272727272727,
  "observationsProcessed": 33,
  "observationsSkipped": 0,
  "eventsCreated": 24,
  "eventsJoined": 9,
  "victimsCreated": 20,
  "ms": 326
}

Stage 4, out loud. The collapse ratio prints on every run, because a rule change that quietly merges more is what you want to notice the same day. Here 33 observations became 24 events: 9 joined an existing event, a reduction of 27%. Real output of the resolve command on the seeded demo database.

A look at the code

The join key, rewritten to stand alone. Decoration is thrown away until 2 spellings of one company produce the same string:

const LEGAL_SUFFIX =
  /\b(incorporated|corporation|limited|inc|llc|llp|pllc|ltd|gmbh|corp|bv|plc|srl|pty)\b/g;

const MIN_KEY_LENGTH = 4;   // anything shorter is an acronym, and acronyms collide

function joinKey(name: string): string | null {
  const key = name
    .normalize("NFKC")
    .toLowerCase()
    .replace(/^www\./, "")
    .replace(LEGAL_SUFFIX, " ")
    .replace(/[^a-z0-9]+/g, "");      // punctuation, spaces, everything
  return key.length >= MIN_KEY_LENGTH ? key : null;
}

joinKey("Acme Industries, Inc.");     // 'acmeindustries'
joinKey("ACME INDUSTRIES LIMITED");   // 'acmeindustries'
joinKey("AIL");                       // null — too short to join on

Three words are missing from that suffix list: co, group and sa. Stripping them was measured and rejected — between them they collided nearly 200 pairs of organisations that merely shared a first word.

Two records agreeing on a victim is not yet one incident. The merge test needs a date too, and how far apart the dates may be depends on how strong the match was:

const DOMAIN_MATCH_WINDOW_DAYS = 90;   // domain + actor: a strong key
const NAME_MATCH_WINDOW_DAYS   = 30;   // name + actor: riskier, so narrower

function isSameEvent(record, existingEvent): boolean {
  if (!record.eventDate) return false;                    // no usable date, never merges
  if (existingEvent.victimId !== record.victimId) return false;
  if (existingEvent.actorId !== record.actorId) return false;

  const matchedOnDomain = Boolean(record.domain);
  if (!matchedOnDomain && (record.nameIsMasked || !record.nameKey)) return false;

  const gapDays =
    Math.abs(+existingEvent.eventDate - +record.eventDate) / 86_400_000;
  return gapDays <= (matchedOnDomain ? DOMAIN_MATCH_WINDOW_DAYS : NAME_MATCH_WINDOW_DAYS);
}

A record with a masked name and no domain never joins on the name. Sources restating the same masked title agree about the mask, not about the victim.

How far apart 2 reports of the same victim and actor can be and still merge into one event, on a scale from 0 to 90 days. Matched on domain and actor, the window is 90 days. Matched on name and actor, it is 30 days. A masked name with no domain never merges on the name, and a record with no usable date never merges at all. — **The weaker the match, the shorter the window.** The 2 windows are the constants in the code above; the 2 refusals are its early returns.

What it can’t do

The dates are imprecise, and the corpus says so

Of all events, 73% sit at exactly midnight and 11% on the first of a month, because the upstream dates are day- or month-precision and nothing records which. A weekly report covering the first to the seventh over-counts, every time. That figure is published alongside the data rather than buried, because a date column that looks precise and is not is the easiest way to publish something wrong out of this.

More sources is not more certain

Of all events, 60% have exactly one source. Of the rest, many carry a description that is byte-identical across 2 of their sources — the catalogues copy each other. When 5 of them see one leak-site post, that is evidence the post exists, not 5 confirmations that the breach happened, so agreement between mirrors is capped below the top confidence band.

It splits when it cannot tell

Short names, masked names and records with no usable date are refused a join rather than guessed at, so some real duplicates survive as separate events. Every declined merge is recorded as a link to the event it nearly joined, which makes the split visible — but it is still a split, and a split inflates the total, so the count is a ceiling rather than a truth.

Three shares of all events, each drawn against a full bar of 100%: 73% of events sit at exactly midnight, 11% sit on the first of a month, and 60% have exactly one source. The bars are separate shares and do not add up. — **The caveats, at the size they really are.** Most dates are less precise than they look, and most events rest on a single source.

The caveats ship with the numbers. Anything that only appears in the documentation gets quoted without it. Three sections of one page, shown stacked with the chart and filters between them left out. Real data, measured on 11 October 2026: later than the rebuild behind this post’s headline figures, so every count is higher, 17 sources are active, and the single-source share reads 64% rather than 60%. The date-precision shares have not moved. The gap between 64% and 75% is the mirroring, because sources that copy each other word for word count once. The observations line carries its own warning: records collected since the last resolve belong to no event yet.

And one that does not fit in a box: none of this verifies anything. A canonical event is an accurate summary of what 18 catalogues said. Whether the incident happened as claimed is a separate question, and it is still yours.

Where this sits in the series

This is stage 4 of the 6-stage spine carrying almost the whole weight of a system. Collection is ordinary and the interface is a search box; the value is entirely in the narrowing.

It is also the argument for stage 2. The first workflow post says normalising feels like ceremony with one source and becomes unavoidable with 5. At 18 it is not a stage. It is the product.

What the collapse buys you. One row per incident, with its sources as a column rather than a research task. Real data: one widely reported victim, 6 of its own events from 2022 to 2024, beside 5 organisations whose records mention it. The July 2024 breach carries the repeat-victim flag. The 2 ransomhub claims of 8 April 2024 stay separate events, one under a longer name with no domain: the false split the tool prefers to a false merge.

From a search to the records underneath. Real data from my own aggregator, stitched from screenshots taken at each step rather than recorded. Source names and account-specific panels are hidden.

Storing and searching what you collect is the method behind this post: the same resolution problem taught generally, with the observation-versus-interpretation split underneath it. If this post is the case, that one is the argument.