Alerting on Geocoding Match Rate Drops

TL;DR: Compare the current match rate against a rolling baseline rather than a fixed constant, require a minimum volume before the comparison is trusted, evaluate per market rather than in aggregate, and demand two consecutive breaches before paging. Those four rules turn the most useful signal in a geocoding pipeline into one people still trust after six months. The metric design is in monitoring and alerting for geocoding pipelines.

Why a Fixed Threshold Fails

The obvious alert — match rate below 95% — is wrong in both directions. It fires constantly in markets where 88% is excellent, and it never fires in markets where a drop from 99.4% to 96% represents a serious regression. A single constant cannot describe a metric whose healthy value differs by an order of magnitude between segments.

One constant cannot serve two markets Two match-rate series over time. The first market runs steadily at 88 percent, permanently below a fixed 95 percent threshold, producing constant false alarms. The second runs at 99 percent and drops to 96 percent, a serious regression that stays above the fixed threshold and is never detected. A rolling baseline detects the second and ignores the first. fixed threshold, 95% market A — healthy at 99% real regression, never alerts market B — healthy at 88% permanently "alerting" — nobody looks any more A baseline built from each series' own recent history detects the drop in A and stays silent for B.

The lower line is how alerting dies. A signal that is permanently breached is muted within a week, and the mute usually covers the whole rule rather than the one market — so when a real regression appears in a different segment, nothing fires.

The Rule

groups:
  - name: geocoding-quality
    interval: 5m
    rules:
      # Baseline: the same metric averaged over the last seven days.
      - record: geocode:match_rate:baseline7d
        expr: avg_over_time(geocode:match_rate:5m[7d])

      - alert: GeocodeMatchRateDrop
        expr: |
          (
            geocode:match_rate:5m
            < (geocode:match_rate:baseline7d - 0.03)
          )
          and
          (
            sum by (market) (rate(geocode_lookups_total[5m])) > 0.5
          )
        for: 10m
        labels:
          severity: page
        annotations:
          summary: >-
            Match rate in {{ $labels.market }} is {{ $value | humanizePercentage }},
            more than 3 points below its 7-day baseline.
          runbook: https://internal/runbooks/geocode-match-rate

Three of the four rules are visible in that expression. The comparison is against a seven-day rolling average rather than a constant, the and clause enforces a minimum request rate so quiet markets do not alert on statistical noise, and for: 10m requires the condition to persist across two evaluation intervals before it pages.

Choosing the Window and the Margin

Parameter Too small Too large Reasonable
evaluation window noisy; fires on single bad minutes slow to detect a real drop 5 minutes for streaming, per-run for batch
baseline period absorbs the regression it should detect slow to adapt to genuine improvements 7 days
margin below baseline constant false alarms misses meaningful regressions 2–3 percentage points
persistence (for) pages on transients delays a genuine page two evaluation intervals

The baseline period has a subtlety worth understanding: a baseline that is too short learns the regression. If the average is taken over an hour and the match rate degrades gradually over a day, the baseline follows it down and the alert never fires. Seven days is long enough to resist that and short enough to accommodate a deliberate improvement within a week.

Volume Guards Prevent Most False Alarms

Ratios are unstable at small denominators. A market with four lookups in a five-minute window can show a match rate of 75% because one address failed, and that is not a signal about anything.

Ratio stability against volume A scatter of observed match rates against the number of lookups in the window. At low volumes the observed rates scatter widely around the true value. Above a threshold of roughly one hundred lookups per window the scatter narrows sharply. A vertical line marks the volume below which the alert is suppressed. true rate suppress alerts left of this line few lookups per window many lookups per window

Set the volume guard from your own data rather than from a rule of thumb: plot observed rates against window volume for a week and find the point where the scatter narrows. In most pipelines that lands somewhere between fifty and two hundred lookups per window.

Making the Alert Useful at 3 a.m.

An alert that says only “match rate down” hands the responder a research project. Attaching the three most likely explanations to the payload turns it into a triage that takes a minute.

The provider mix is the first: if the share of records answered by the fallback provider jumped in the same window, the primary is degraded and the chain is doing its job, which is a different situation from a genuine quality regression. The cache hit rate is second: a collapse there points at a key or deployment change rather than at anything provider-side. The recent deploy list is third, because a normalisation change is the single most common cause of a step drop.

All three are cheap to include as annotations rendered from the same metrics, and their presence changes the first action from “start looking” to “confirm or rule out”. That is usually the difference between a fifteen-minute incident and a two-hour one.

Batch Pipelines Need a Different Shape

Everything above assumes a stream of lookups arriving continuously. A nightly batch produces one measurement per run, and a five-minute evaluation window is meaningless against it — the metric is either absent or a single value.

For batch, compare each run against the previous runs rather than against a time window. The baseline becomes the median of the last seven runs, the volume guard becomes a minimum record count for the run, and the persistence requirement becomes two consecutive runs rather than two evaluation intervals. The logic is identical; only the unit of comparison changes.

The one genuinely new consideration is the missing run. A streaming pipeline that stops producing metrics is obvious within minutes; a batch that never starts produces no measurement at all, and an alert defined only on the metric’s value will never fire. Add a staleness check — no batch result recorded in the last twenty-six hours — as a separate rule, and make it a page, because a batch that silently did not run is usually worse than one that ran badly.

Both rules should carry the run identifier in their annotations. Being able to jump straight from the alert to the run’s logs removes the first and most tedious step of any batch investigation, and it is a single label to add at push time.

The Four Rules Together

Four filters between a metric and a page Four stacked filters. The rolling baseline removes false alarms from markets whose healthy rate is low. The volume guard removes noise from small denominators. Segmentation prevents a small market being averaged away. The persistence requirement removes single-window transients. 1 · rolling baseline — removes markets whose healthy rate is simply lower 2 · volume guard — removes ratios computed from a handful of lookups 3 · segmentation — stops a small market being hidden inside an average 4 · persistence — removes the single bad window that fixes itself

Edge Cases and Failure Modes

Deliberate changes that move the metric. Adding a market with poor coverage lowers the aggregate legitimately. Segmenting by market handles it; a global alert would fire for a successful launch.

Baselines poisoned by an outage. A day-long incident inside the seven-day window drags the baseline down and makes the alert insensitive for a week afterwards. Excluding known incident windows from the baseline is possible in most systems and is worth doing if incidents are frequent enough to matter.

Alerting on the corpus rather than production. Corpus results are a scheduled measurement, not a live signal, and they belong in the CI gate. Alerting on both without distinguishing them produces two pages for one cause.

Silent segment disappearance. If a market stops sending traffic entirely, its ratio becomes undefined rather than low, and the alert never fires. A separate absence check — no lookups from a market that normally has some — catches it.

Integration Note

This alert reads the recording rules defined in exporting geocoding pipeline metrics to Prometheus, and it is the live counterpart to the offline check performed by the regression gate. The two are complementary: the gate stops a bad change from shipping, and this alert catches the drifts that no code change caused — a provider model update, a reference-data release, or a shift in the addresses your customers are sending.

Whichever shape you use, write the runbook link into the annotation. An alert that arrives without one is an invitation to improvise at exactly the moment improvisation is least useful.

A runbook that names the three most likely causes, in order, is usually enough.