Multilingual monitoring at production scale

The Analyst's Stack · Showcase

Other people’s shifts start with this tool, and that single fact is what put a queue under every one of its stages. It collects in 3 languages, enriches what it collects, and holds all of it behind a review gate.

Why I built it

A newsroom verification desk works in more languages than any one person on it reads. This one runs on Arabic, English and Hebrew, and most of the people do 2 of the 3.

The daily failure that produces is not the one people expect. It is not that things go untranslated. It is that the language you do not read becomes the language you stop checking, so a story reaches you through the 2 sources you can skim rather than the 8 that were carrying it.

This system is larger than everything else in this series for one reason: the moment a rota depends on a tool, it acquires obligations a personal script does not have. It has to be restartable. It has to say when it has stopped. It has to be wrong in ways somebody can see.

What it solves

Sources are visited on a schedule instead of by whoever remembers. A tick every 30 seconds works out which are due: 15 minutes by default, 2 for anything urgent, an hour for a source quarantined after repeated failures. A source that keeps breaking gets quieter, not louder.

A bar chart of how often a source is visited, on one scale from 0 to 60 minutes. An urgent source is visited every 2 minutes, a default source every 15 minutes, and a source quarantined after repeated failures every 60 minutes. A scan tick every 30 seconds decides which sources are due. — **Failure buys a source less attention, not more.** A quarantined source is visited 30 times less often than an urgent one, so a broken feed cannot crowd out the working ones.

Everything collected is extracted, language-detected, translated, classified, geotagged and scored — and then it waits. Nothing an AI step produced counts until a person approves it, and the search index knows the difference: approved translations and pending drafts live in separate fields, and only a reviewer’s search reaches the drafts, because an unapproved draft anyone can find by searching is an unapproved draft that has been published.

Where the shift works. Three scripts in one list, each row carrying the language the system thinks it is in. The top row is a Hebrew article shown under its Arabic machine draft, which the Arabic interface prefers whenever one exists. A separate local demo of the app, seeded with invented articles.

How it works

The spine, with a queue under each stage. Same 6 as the foundation post — but here each one is a named queue with its own worker and its own concurrency.

Collect. A scan tick decides which sources are due and turns each into a discovery job. Feeds, sitemaps and social sources end up in the same queue.

Normalize. Fetch, strip, extract the body through a small Python service, then answer the question everything downstream depends on: what language is this. That answer is written onto the row.

Store. PostgreSQL for the articles, a search index for retrieval. A translation is a row with a status, not a column on the article.

Analyze. Translate, classify, geotag, summarize, score — each its own job.

Alert. Email out of band, plus webhooks and in-app notifications, all throttled.

Present. A bilingual dashboard whose default text direction is right-to-left.

The gate. A translation exists, is visible to a reviewer, and is not yet part of the record. The translation’s own approve button sits in the reviewer’s editor, open here. A separate local demo seeded with invented articles; the “machine” draft was written for the demo, and no model was called.

One query, 3 scripts. An Arabic search reaches an English article through its approved Arabic translation. Neither English headline contains the word; both match through approved translations. The search ran as a reviewer, whose searches also reach pending drafts, which is how the urgent Hebrew row appears. A separate local demo seeded with invented articles.

How I built it

TypeScript on Bun: an HTTP API, a worker process and a dashboard, with PostgreSQL, Redis, a search cluster and a small Python extraction service beside them. The worker is the interesting half — 17 queues, each with its own concurrency limit, from 4 parallel fetchers down to 1 for anything that drives a browser.

Every stage is a queue, and that is what makes it restartable. Job identifiers are derived from the article, so handing a stage the same article twice is deduplicated rather than duplicated, and re-running one stage on one article is a one-line enqueue. When an operator throws the stop switch mid-pipeline, jobs are re-enqueued rather than dropped, with a delay that starts at 5 minutes, doubles up to 30, and carries jitter so 1,000 paused jobs do not wake in the same second. No article is lost, only slowed.

The other 4 queues never notice. A provider outage surfaces as one queue’s backlog growing, which is also the only place anyone has to look to find it.

Right-to-left is a data question, not a CSS question. I treated it as styling, and that was wrong. Block direction has to come from the article’s own language, not the interface language — a Hebrew article read by an Arabic-speaking analyst is still a Hebrew article — and mixed text needs per-fragment bidi handling (the rules deciding the visual order of mixed-direction text) so a Latin source name inside an Arabic sentence does not reorder the sentence around it.

What I missed for too long: article bodies carried a direction but no language tag. Roughly a third of the stored corpus is Hebrew, and none of it was announced as Hebrew to the browser. Screen readers read it in the Arabic voice; font fallback resolved against the wrong language. A test now fails if any element declares a direction without a language.

The hard part was language detection failing quietly. The first detector was built from exactly 3 languages, and a detector can only return a language it was constructed with — so a French article came back labelled as one of the 3, and no downstream work could have fixed it, because a fourth language was not observable. Widening the candidate set to 34 codes creates the mirror problem: Arabic, Persian and Urdu share a script, so Arabic text can now be answered fa, and every Arabic filter built on the stored language quietly loses it.

What I would do differently is operations. The queue library keeps finished jobs forever by default and the extraction queue’s payloads carry raw page HTML, so the deployed cache reached 2.74 GB of orphaned job records with every queue sitting empty. And the ladder that pauses AI work on a provider error was fully built while the half that tells a human was not, so a quota storm at three in the morning halted enrichment and waited until someone opened the admin console.

And there is the entity-grouping step that filled a bucket labelled names with things that were not names, for months, while looking fine every time I checked it by eye. That one is worth a post on its own, and it has one: does your AI step actually work? counts it properly.

A look at the code

Language detection, rewritten to stand alone — a margin decides when a fourth language is believed over the 3 the desk runs on:

CORE = ("ar", "en", "he")   # the three the desk actually runs on
MARGIN = 0.30               # how much better a fourth language must look

def pick_language(scores):
    """`scores`: (code, confidence) pairs over every candidate, best first."""
    if not scores or scores[0][1] <= 0:
        return None                  # empty text scores every language zero
    top_code, top_value = scores[0]
    if top_code in CORE:
        return top_code
    best_core_value = max((v for c, v in scores if c in CORE), default=0.0)
    if top_value - best_core_value >= MARGIN:
        return top_code              # the evidence is one-sided; record it
    return next((c for c, v in scores if c in CORE), None)

pick_language([("fa", 0.55), ("ar", 0.41), ("en", 0.02)])   # -> 'ar'
pick_language([("fr", 0.93), ("en", 0.21), ("ar", 0.01)])   # -> 'fr'
pick_language([("ar", 0.00), ("en", 0.00)])                 # -> None

Two language-detection results from the code example above, drawn as score bars on a scale from 0 to 1. In the first, Persian scores 0.55 and Arabic, the best of the 3 core languages, scores 0.41; the gap of 0.14 is under the 0.30 margin, so the article is recorded as Arabic. In the second, French scores 0.93 and English, the best core language, scores 0.21; the gap of 0.72 clears the margin, so the article is recorded as French. — **The margin, drawn.** The scores are the ones in the example calls above: a close call goes to a core language, and only a one-sided result adds a fourth.

And the queue setup — a few lines of configuration, and one of the more consequential things in the system:

import { Queue } from "bullmq";

// The library's default is "keep every finished job, forever". Nothing here
// ever reads one back, and the extract queue's payloads carry raw page HTML.
const retention = {
  removeOnComplete: { count: 200 },
  removeOnFail: { count: 2000 },  // failures are rarer, smaller, and worth keeping
};

const stages = ["discover", "fetch", "extract", "enrich",
                "snapshot", "screenshot", "fullcopy", "media"];

// `connection` is the Redis host/port object.
const queues = new Map(
  stages.map((s) => [s, new Queue(s, { connection, defaultJobOptions: retention })]),
);

// A stable job id is what makes a re-run safe: ask twice, run once.
await queues.get("enrich").add("enrich", { articleId }, { jobId: `enrich-${articleId}` });

What it can’t do

Detection is a guess, with a deliberate bias

A genuinely French or Persian article is recorded as one of the 3 core languages unless the evidence is decisive. That trade is chosen: a wrong fourth language silently removes an article from the filters the desk lives in. Text the detector cannot read at all is stored with no language rather than a guess.

Nothing a model produced is evidence

Translations, classifications and summaries arrive as drafts and stay drafts until a person approves them. A rejected translation is dropped from search entirely rather than filed as a draft, because a refusal is a decision.

It is infrastructure, not a tool

A database, a cache, a search cluster, an extraction service, an API and a worker. That is the real cost of the queue, the gate and the 3 languages — and it is why the one-file watcher earlier in this series is still right for one source.

The Queues panel of the jobs board: 17 named queues, from scan, discover, fetch, extract and enrich down to backup, source-health and llm-batch, each with counts of waiting, active, delayed, completed and failed jobs. fetch has 7 waiting and 3 failed; extract and snapshot have 1 failed each; 8 queues show completed jobs; nothing is active; 9 queues are at zero in every column.

The screen that says it is alive. A pipeline nobody can see the state of is a pipeline nobody trusts on a Sunday. One queue has a backlog of 7 and 3 failures; 9 sit idle. Real queue counts from a separate local demo seeded with invented articles and placeholder jobs; the 64 completed fetch jobs are among the placeholders, and no page was requested.

Where this sits in the series

This is the 6-stage spine carrying weight. Put it beside monitoring at scale: same shape, same review discipline, one structural difference. There, the queue is a table and one worker drains it. Here, every stage has its own queue and its own way of failing — the price of being the thing a shift starts its morning with.

Two later posts go deeper into pieces this one only names. Does your AI step actually work? tells the full story of that extraction step, and Storing and searching what you collect covers the schema question underneath all of it.