Scraping and mapping a public alerts feed

A public authority publishes every alert it issues, and offers no way to ask anything of the 12 months behind it. This walks that feed one record at a time and turns it into 2 map layers and a set of charts.

Why I built it

A newsroom verification desk had a question about a public alerts feed — the kind a public authority publishes so anyone can see what has just been issued, as it happens. Not about this week: over the past 12 months, how many alerts, in which cities, of which category, and did the pattern move.

The feed answers none of that. It shows you the newest item and then forgets. No date filter, no export, no bulk download, and a rate limit that ends the conversation if you ask too quickly. What it does have is one useful property: every record ever filed sits at its own sequential number, and you can ask for one at a time. That turns a research question into a collection problem, which is an easier kind of problem to have.

What it solves

One run turns a range of record numbers into the raw records on disk, one flat table, 2 map layers — counts by city, and the same counts dissolved into the larger areas those cities sit inside — and charts of each category month by month.

The number worth quoting is the unglamorous one: 37, how many variant spellings of city names the normalize step collapses before any of the mapping works. The feed and the boundary file disagree about hyphens, prefixes and plain typos, and a join (matching rows in one table to shapes in another by a shared key) does not fail loudly on a name that does not match. It drops the row.

Collection has its own 2 numbers: 50 records in flight at once, 3 requests per exit before that exit rotates out.

How it works

Collect walks a range of record numbers, 50 at a time, each request leaving through a rotating exit. A 404 means nothing was filed there.

Normalize flattens each record into rows — one row per city named in one alert — applying the name-fix table on the way through.

Store writes each raw record to disk as JSON before anything touches it, then the flat table as CSV. The raw copy makes a normalize bug cost a re-run, not a re-collection.

Map joins the counts to a boundary file, repairs whatever geometry needs repairing, and writes 2 GeoJSON layers (plain text holding shapes and their attributes; every mapping tool reads it) — one at city level, one dissolved to area level.

Chart groups the same table by month and by category, and renders bar and line charts as images.

The collection loop from further down, run against a synthetic feed on my own machine: 300 record numbers, 8 of them withdrawn, and one rate limit planted to see what happens.

$ python3 walk_demo.py 1000 1299
walking ids 1000-1299 on a local synthetic feed: 50 in flight, 3 requests per exit
id 1037  404  nothing filed, walk continues
id 1048  404  nothing filed, walk continues
id 1077  404  nothing filed, walk continues
id 1150  429  rate limited: wait 5 s, next exit
id 1165  404  nothing filed, walk continues
id 1187  404  nothing filed, walk continues
id 1202  404  nothing filed, walk continues
id 1274  404  nothing filed, walk continues
id 1024  404  nothing filed, walk continues
done in 5.2 s
  records saved          292
  not filed (404)          8
  rate limited, retried    1
  requests per exit     exit-a 75  exit-b 76  exit-c 75  exit-d 75

How I built it

Python, because the mapping half decides the language: geopandas and shapely for geometry, pandas for the table, matplotlib for the charts, httpx and asyncio for fetching.

The first version fetched one record at a time with a polite sleep between requests. It worked, and for a few thousand records it would have run overnight. Async fixed that and brought the real difficulty with it: a semaphore (a counter that stops more than N things running at once) caps you at 50 in flight, and 50 in flight from one address is exactly the traffic shape a rate limiter exists to catch. So the exits rotate on a budget — 3 requests each, then move on.

The stop condition was the other nuisance. Nothing tells you where the range ends, so a 404 is the signal — and a 404 also happens mid-range, where a record was withdrawn. The version that treats the first one as the end is wrong in a way that looks like success.

Then the geometry. Boundary files from public sources are full of polygons that are almost valid: a ring that crosses itself, a sliver with zero area, points stored in the wrong winding order. None of it is visible on a map. All of it makes geopandas refuse to dissolve, measure or join.

What the 2 map layers look like, with invented shapes and counts standing in for real ones:

Two things I would change. The backoff is a set of fixed sleeps, when the server is already saying how long to wait in its own response headers. And the 37 name fixes live in a dictionary inside the code, invisible to anyone not reading it; they belong in a data file beside the boundary file they exist to agree with.

The 2 chart shapes the last step writes, drawn here with invented numbers in them:

A look at the code

The collection loop, standing alone. One pool of exits, one semaphore, and a per-exit budget that rotates the pool instead of sleeping.

import asyncio
from collections import deque
import httpx

FEED = "https://example.test/alerts-history/id"
PER_EXIT = 3       # requests an exit serves before it is rotated out

async def fetch(record_id, pool, gate):
    """One record as a dict, or None if nothing is filed under that id."""
    async with gate:
        for _ in range(3):
            client, used = pool[0]
            pool[0] = (client, used + 1) if used + 1 < PER_EXIT else (client, 0)
            if used + 1 >= PER_EXIT:
                pool.rotate(-1)         # this exit has had its turn
            try:
                r = await client.get(f"{FEED}/{record_id}")
            except httpx.TransportError:
                pool.rotate(-1)
                continue
            if r.status_code in (200, 404):
                return r.json() if r.status_code == 200 else None  # 404: nothing filed
            await asyncio.sleep(5)      # rate limited, or a server error
            pool.rotate(-1)
    return None

async def walk(first, last, exits):
    pool = deque((httpx.AsyncClient(proxy=u, timeout=20), 0) for u in exits)
    gate = asyncio.Semaphore(50)        # records in flight at once
    return await asyncio.gather(*(fetch(i, pool, gate) for i in range(first, last + 1)))

And the mapping step. counts is a 2-column table of city name and alert count; the rest is repair and 2 writes.

import geopandas as gpd

def repaired(gdf):
    """Drop empty shapes, rebuild the broken ones, say how many."""
    gdf = gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()
    broken = ~gdf.geometry.is_valid
    if broken.any():
        # buffer(0) rebuilds a polygon from its own edges. It fixes most
        # self-intersections and silently reshapes the rest — so count them.
        print(f"repairing {int(broken.sum())} invalid geometries")
        gdf.loc[broken, "geometry"] = gdf.loc[broken, "geometry"].buffer(0)
    return gdf[gdf.geometry.is_valid]

cities = repaired(gpd.read_file("city_boundaries.geojson"))
layer = cities.merge(counts, on="city", how="left").fillna({"alerts": 0})

unmatched = set(counts["city"]) - set(cities["city"])
if unmatched:
    print(f"{len(unmatched)} names have no polygon: {sorted(unmatched)}")

layer.to_file("alerts_by_city.geojson", driver="GeoJSON")
layer.dissolve(by="area", aggfunc={"alerts": "sum"}).to_file(
    "alerts_by_area.geojson", driver="GeoJSON"
)

The lines that print unmatched are what I would keep if I had to delete everything else. A silent left join is how you publish a map with 37 holes in it.

What it can’t do

There is no alerting stage here at all, and that is deliberate. This answers a question about the past; it does not watch the present.

Where this sits in the series

This is the spine from what automation actually means for an OSINT analyst, with the Analyze stage kept deliberately dumb — no model, no classification, just grouping and counting — and the Present stage doing all the work.

The rotation and the 3-requests-per-exit budget are the practical version of OPSEC for automation. Rotation here is not about hiding. It is about not being the traffic that makes a public feed decide to stop being public.

Two later posts pick up threads from this one. Storing and searching what you collect covers what to do when the same place arrives under 3 spellings from 2 sources, which is the half of normalization this post only gestures at. Does your AI step actually work? is the one to read before trusting the 37: a hand-built fix table is a recall problem, and recall is measurable rather than assumed.