Building Regression Corpora for Geocoding Accuracy

A regression corpus is the single most effective guard against silent accuracy loss in an automated geocoding pipeline. It is a fixed, versioned set of addresses whose correct coordinates are already known, run against your providers on a schedule so that any drop in match-rate or rise in positional error is caught before it reaches production. This topic sits within Accuracy Validation & CI/CD Sync, and it is the measurement backbone that the validating geocoding accuracy and confidence scoring workflow and the CI/CD automation for spatial enrichment both depend on.

Providers change constantly. A vendor refreshes its underlying address database, your team ships a new normalization rule, a fallback weight shifts, and match quality moves — usually invisibly, because a geocoding API almost never returns an error when it degrades; it returns a plausible-looking wrong coordinate. Without a fixed reference set scored against known truth, you discover the regression when a downstream route calculation or spatial join breaks weeks later. The corpus turns that invisible drift into a numeric signal you can gate on.

---

Prerequisites

• Python 3.8+ with numpy and pandas installed, plus your geocoding client library
• A source frame of real addresses (production logs, a customer table, or an open address dataset) large enough to sample from
• At least one authoritative source of known coordinates: a national address register, cadastral/parcel data, or GPS-surveyed points
• A metric definition agreed with downstream consumers: the hit distance threshold and the minimum acceptable match-rate per address class
• A version-control location for the corpus file and its baseline metrics (git for small corpora, DVC or object storage for large ones)

Read the corpus config as the contract for the whole measurement. Everything downstream — the runner, the metrics, the gate — is meaningless unless the hit threshold and the strata are pinned and versioned alongside the data.

Step 1 — Design a Stratified Sampling Frame

A random sample of production addresses is dominated by the formats you already handle well: dense urban street addresses in your primary market. The regressions that hurt happen in the tail — rural routes, PO Boxes, non-Latin scripts, newly built subdivisions, cross-border formats. If those appear once per ten thousand records, a random corpus barely contains them, and a provider can lose them entirely without moving your headline number.

Stratify instead. Partition the population along the axes where provider quality actually varies — country, region or state, and address type — and sample a fixed target count from each cell. This guarantees rare formats are represented regardless of their natural frequency. The full sampler, including a pandas groupby implementation and provenance recording, is covered in building a stratified ground-truth address test set.

Choose strata that (a) you have enough truth data to fill and (b) correspond to a real quality boundary. Country is almost always a stratum because postal conventions and provider coverage change hardest at borders. Address type — street, unit/apartment, PO Box, rural route, intersection — is the second axis because geocoders resolve each class through different pipelines.

Step 2 — Source Authoritative Ground Truth

The corpus is only as trustworthy as its truth column. There are three defensible sources, in descending order of confidence:

1. Surveyed or GPS-captured points — a field team or a delivery telematics feed recording the actual rooftop or entrance coordinate. Highest confidence, most expensive.
2. Authoritative registries — national address files (for example a cadastral parcel centroid or an official address point layer) that publish coordinates as part of the record. Strong confidence for the jurisdictions that maintain them.
3. Consensus geocoding — geocode with three or more independent providers and accept the point only when they agree within a tight radius. Cheapest, but it inherits the shared blind spots of the providers and must never be used to benchmark those same providers.

Record which source produced each truth coordinate. A mixed corpus is fine, but a metric computed across cases of different truth quality must be interpreted with that in mind, and stale-truth quarantine (below) depends on knowing the source.

Step 3 — Define a Versioned Corpus Schema

Store the corpus as one row per case in a columnar format (CSV for review-friendliness, Parquet for scale). The schema below is the reference contract every runner and metric reads.

Version the file itself. Bump a minor version when cases are added, a patch when truth is corrected, and a major version when strata or the hit threshold definition change, because those invalidate comparison against an older baseline. Never edit a case in place without bumping the version — a silent truth edit makes a passing build look like a regression fix and destroys the audit trail.

Step 4 — Run the Corpus and Compute Metrics

The corpus runner loads the cases, geocodes each raw_input through the provider under test, measures the haversine distance from the returned coordinate to the truth coordinate, classifies each case as a hit or miss against its hit_threshold_m, and aggregates match-rate and error percentiles per stratum. Below is a self-contained row-level implementation.

from __future__ import annotations

import csv
import math
from dataclasses import dataclass
from statistics import median
from typing import Callable, Optional

_EARTH_RADIUS_M = 6_371_000.0


def haversine_m(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
    """Great-circle distance in metres between two WGS84 points."""
    p1, p2 = math.radians(lat1), math.radians(lat2)
    dphi = math.radians(lat2 - lat1)
    dlmb = math.radians(lon2 - lon1)
    a = math.sin(dphi / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dlmb / 2) ** 2
    return 2 * _EARTH_RADIUS_M * math.asin(math.sqrt(a))


@dataclass
class CaseResult:
    case_id: str
    stratum: str
    error_m: Optional[float]   # None when the provider returned nothing
    is_hit: bool


# A GeocodeFn takes a raw address and returns (lat, lon) or None on no-match.
GeocodeFn = Callable[[str], Optional[tuple[float, float]]]


def run_corpus(path: str, geocode: GeocodeFn) -> list[CaseResult]:
    """Geocode every case in a corpus CSV and score it against truth.

    Args:
        path:     Path to the corpus CSV following the reference schema.
        geocode:  Callable that geocodes one address string.

    Returns:
        One CaseResult per corpus row.
    """
    results: list[CaseResult] = []
    with open(path, newline="", encoding="utf-8") as fh:
        for row in csv.DictReader(fh):
            stratum = f"{row['country']}/{row['address_type']}"
            threshold = float(row["hit_threshold_m"])
            expect_match = row["expected_outcome"] == "match"

            try:
                point = geocode(row["raw_input"])
            except Exception:  # provider fault: scored as a miss, never crashes the run
                point = None

            if point is None:
                # A no-match is correct for a negative case, a miss otherwise.
                results.append(CaseResult(row["case_id"], stratum, None, not expect_match))
                continue

            if not expect_match:
                # Provider returned a confident point for input that should not resolve.
                results.append(CaseResult(row["case_id"], stratum, None, False))
                continue

            err = haversine_m(float(row["truth_lat"]), float(row["truth_lon"]), point[0], point[1])
            results.append(CaseResult(row["case_id"], stratum, err, err <= threshold))
    return results


def summarize(results: list[CaseResult]) -> dict[str, dict[str, float]]:
    """Aggregate match-rate and error percentiles per stratum."""
    strata: dict[str, list[CaseResult]] = {}
    for r in results:
        strata.setdefault(r.stratum, []).append(r)

    summary: dict[str, dict[str, float]] = {}
    for stratum, rows in strata.items():
        errors = sorted(r.error_m for r in rows if r.error_m is not None)
        hits = sum(1 for r in rows if r.is_hit)
        summary[stratum] = {
            "n": float(len(rows)),
            "match_rate": hits / len(rows),
            "median_error_m": median(errors) if errors else float("nan"),
            "p95_error_m": errors[int(0.95 * (len(errors) - 1))] if errors else float("nan"),
            "coverage": len([r for r in rows if r.error_m is not None]) / len(rows),
        }
    return summary

The metric definitions and how to read each one are broken down in measuring geocoding match-rate in Python.

Vectorized pandas variant

For corpora above a few thousand cases, or when benchmarking several providers at once, aggregate with pandas. The vectorized haversine runs over whole coordinate arrays and the per-stratum rollup is a single groupby.

import numpy as np
import pandas as pd

_EARTH_RADIUS_M = 6_371_000.0


def haversine_series(
    lat1: pd.Series, lon1: pd.Series, lat2: pd.Series, lon2: pd.Series
) -> pd.Series:
    """Vectorized great-circle distance in metres over aligned coordinate Series."""
    p1, p2 = np.radians(lat1), np.radians(lat2)
    dphi = np.radians(lat2 - lat1)
    dlmb = np.radians(lon2 - lon1)
    a = np.sin(dphi / 2) ** 2 + np.cos(p1) * np.cos(p2) * np.sin(dlmb / 2) ** 2
    return 2 * _EARTH_RADIUS_M * np.arcsin(np.sqrt(a))


def score_frame(df: pd.DataFrame) -> pd.DataFrame:
    """Score a corpus DataFrame that already has geocoded got_lat / got_lon columns.

    Expects columns: country, address_type, truth_lat, truth_lon,
    got_lat, got_lon, hit_threshold_m, expected_outcome.
    """
    df = df.copy()
    resolved = df["got_lat"].notna() & df["got_lon"].notna()
    df["error_m"] = np.where(
        resolved,
        haversine_series(df["truth_lat"], df["truth_lon"], df["got_lat"], df["got_lon"]),
        np.nan,
    )
    expect_match = df["expected_outcome"].eq("match")
    df["is_hit"] = np.where(
        expect_match,
        resolved & (df["error_m"] <= df["hit_threshold_m"]),
        ~resolved,  # a negative case is a hit only if nothing resolved
    )
    df["stratum"] = df["country"].str.cat(df["address_type"], sep="/")

    return df.groupby("stratum").agg(
        n=("is_hit", "size"),
        match_rate=("is_hit", "mean"),
        median_error_m=("error_m", "median"),
        p95_error_m=("error_m", lambda s: s.quantile(0.95)),
        coverage=("error_m", lambda s: s.notna().mean()),
    )

Step 5 — Wire the Result into a Regression Gate

Metrics are only a gate when they are compared to a committed baseline and the build fails on a regression. Store the previous release’s per-stratum summary as a JSON baseline in the repository. On each run, load it, diff the new numbers, and exit non-zero when any stratum drops below tolerance.

import json
import sys
from typing import Mapping

# Tolerances are absolute, not relative, so a small stratum can't hide a real drop.
MATCH_RATE_DROP_TOL = 0.02      # allow 2 percentage points of noise
MEDIAN_ERROR_RISE_TOL_M = 10.0  # allow 10 m of median drift


def gate(current: Mapping[str, dict], baseline_path: str) -> int:
    """Fail (return non-zero) if any stratum regressed beyond tolerance."""
    with open(baseline_path, encoding="utf-8") as fh:
        baseline = json.load(fh)

    failures: list[str] = []
    for stratum, cur in current.items():
        base = baseline.get(stratum)
        if base is None:
            continue  # new stratum: nothing to compare against yet
        if base["match_rate"] - cur["match_rate"] > MATCH_RATE_DROP_TOL:
            failures.append(
                f"{stratum}: match-rate {base['match_rate']:.3f} -> {cur['match_rate']:.3f}"
            )
        if cur["median_error_m"] - base["median_error_m"] > MEDIAN_ERROR_RISE_TOL_M:
            failures.append(
                f"{stratum}: median error {base['median_error_m']:.1f} -> {cur['median_error_m']:.1f} m"
            )

    for line in failures:
        print(f"REGRESSION {line}", file=sys.stderr)
    return 1 if failures else 0

Run this as a step in the workflow described in automating CI/CD sync for spatial enrichment. The gate turns the corpus from a report you occasionally read into a hard stop that blocks a merge, which is the only form that reliably prevents accuracy regressions from shipping. When comparing several vendors rather than tracking one over time, the same runner feeds the head-to-head methodology in comparing geocoding accuracy across providers.

Corpus Field Reference

Edge Cases

Stale Ground Truth

Truth coordinates decay. A rooftop is rebuilt, a parcel is subdivided, a registry corrects an error, and the corpus point silently becomes the wrong answer. The symptom is a case whose error jumps sharply across a build for a provider you did not change. Treat a sudden large error on a previously-passing case as suspect truth, not confirmed regression: quarantine it, route it to human review against a fresh authoritative source, and only then either correct the truth (patch version bump) or confirm the regression. Never let a single build silently rewrite truth.

Sampling Bias

A corpus stratified only on country still hides bias if, within a country, all cases come from one metropolitan source. The provider looks perfect nationally while failing rural formats that never entered the sample. Audit the intra-stratum distribution — plot cases per region and per address type inside each country stratum — and backfill thin cells. The corpus measures exactly what it contains and nothing else.

Ambiguous Truth

Some addresses genuinely have no single correct coordinate: a long rural driveway, a large campus with one street address, a building with entrances on two streets. A tight hit_threshold_m will fail these unfairly. Either widen the threshold for that specific case to reflect real ambiguity, or move it into a coarser truth_precision class scored at street level. Do not delete it — ambiguous addresses are exactly where providers diverge and where the corpus earns its value.

Coordinate Reference Mismatch

Truth from a national registry may be published in a projected national grid rather than WGS84 lat/lon. Reproject to WGS84 before storing truth_lat/truth_lon, and assert the values fall in [-90, 90] and [-180, 180] at load time. A silent projection mismatch produces uniform large errors that look like a total provider failure.

Performance and Vectorization

The scoring math is trivially cheap; the cost is the geocoding calls. Cache each provider response keyed by a hash of the normalized input so that re-running the corpus after a code change that does not touch a given provider costs nothing. Pair this with the rate limiting strategies for batch processing so a full corpus run does not exhaust a daily quota.

Troubleshooting

Match-Rate Collapses Across Every Stratum at Once

A uniform collapse is almost never a provider regression — it is a harness bug. Check for a coordinate order swap (lat/lon transposed), a reprojection that was dropped, or a corpus version that changed hit_threshold_m. A genuine provider regression shows up in a subset of strata; a global collapse points at your own code.

One Stratum Is Always Zero

If a stratum reports match_rate = 0.0 on every run, the geocoder likely cannot parse that address class at all — commonly PO Boxes or rural routes routed to a street-level geocoder. This is a real finding, but it belongs in an expected-outcome baseline rather than a hard-fail, or the gate blocks every build. Set those cases’ hit_threshold_m and expected_outcome to reflect the achievable precision.

The Gate Passes but Production Accuracy Fell

The corpus does not represent production traffic. Compare the strata distribution of the corpus against a recent production sample; if production has shifted toward a format the corpus barely covers, the corpus is stale. Re-sample from current production logs and bump the corpus version.

Median Error Is Fine but p95 Exploded

The provider degraded on a tail of hard cases while the bulk stayed accurate — often a specific region or a newly ambiguous format. The median hides it; the p95 catches it. Always gate on both a central and a tail statistic, and drill into the specific case_id values driving the p95.

FAQ

How large should a geocoding regression corpus be?

Size the corpus by strata, not by a single global number. Aim for at least 30 to 50 cases per stratum so a per-stratum match-rate has a usable confidence interval, then sum across strata. A corpus of 1,500 to 5,000 carefully stratified cases catches far more regressions than 50,000 random urban addresses, because the random sample under-represents the rural, PO Box, and non-Latin formats where providers actually drift.

What match distance counts as a hit?

Set the threshold from the downstream use, not a universal constant. Rooftop-dependent logistics typically treat a haversine error under 25 metres as a hit; parcel-level analytics tolerate 100 metres; regional aggregation tolerates a postal-centroid match in the hundreds of metres. Record the threshold in the corpus config so a metric is always interpretable against the tolerance it was scored under.

How do I stop stale ground truth from failing good geocodes?

Attach a truth_captured_date and a truth_source to every case, and treat large sudden errors as suspect truth rather than provider regression until reviewed. When a provider moves a rooftop closer to a rebuilt parcel, the corpus point may be the stale value. Quarantine cases whose error jumps across a build and route them to human review before updating the baseline.

Should the corpus include ambiguous or unresolvable addresses?

Yes, but tag them as a distinct stratum with an expected outcome of no-match or low-confidence. A corpus that only contains clean, resolvable addresses cannot detect a provider that silently starts returning confident coordinates for garbage input. Negative cases guard the precision side of the metric, not just the recall side.

How often should the corpus run?

Run it on every change to normalization or routing code, on every provider weight update, and on a nightly schedule to catch silent provider-side data changes that no code commit would trigger. The nightly run is what surfaces upstream drift; the per-commit run is what blocks a regression before it merges.

---

Related

• Validating geocoding accuracy and confidence scoring — per-result confidence scoring and outlier detection that feeds the hit/miss classification used here.
• Automating CI/CD sync for spatial enrichment — the pipeline that runs the corpus gate on every commit and schedules the nightly drift check.
• Building a stratified ground-truth address test set — choosing strata, target counts, the CSV/JSON schema, and a pandas sampler with provenance.
• Measuring geocoding match-rate in Python — the metric functions for hit rate, median and p95 haversine error, and coverage.
• Comparing geocoding accuracy across providers — using the same corpus to benchmark multiple vendors head-to-head rather than tracking one over time.