Building a Stratified Ground-Truth Address Test Set

TL;DR: Partition a source frame of real addresses by country, region, and address type, draw a fixed target count from each cell with a pandas groupby().sample(), attach known-good coordinates plus a provenance tag to every row, and version the result. This assembles the stratified ground-truth set that the regression corpus workflow scores against.

A ground-truth test set is a table of addresses whose correct coordinates you already know, sampled so that every address class you care about is present in usable numbers. The quality of every downstream accuracy number depends on how this set is assembled, and the single decision that matters most is stratification: drawing a fixed quota from each meaningful cell rather than sampling proportionally from production, where the tail formats that break geocoders barely appear.

Choosing Strata

Stratify on the axes where geocoding quality genuinely changes. Three axes cover almost every pipeline:

• Country (ISO 3166-1 alpha-2). Postal conventions, provider coverage, and script all change hardest at national borders. This is nearly always the primary axis.
• Region (state, province, prefecture). Within a large country, coverage varies between dense metros and rural areas; a region axis stops one city dominating the sample.
• Address type (street, unit, po_box, rural_route, intersection, negative). Geocoders resolve each class through a different internal pipeline, so each needs its own cell.

The cross-product of these axes defines the strata. Keep only cells that correspond to a real quality boundary and that you can fill with truth data — an empty cell measures nothing. Add a negative address-type cell of inputs that should not resolve (malformed strings, fictional streets); it is the only way to catch a provider that starts confidently returning coordinates for garbage.

Target Count per Stratum

Assign a fixed target to each cell rather than a global fraction. A per-stratum match-rate needs enough cases to be stable: 30 is the practical floor for a rough rate, 50 or more gives a tighter interval. Weight the target up for strata that are both high-volume in production and high-risk.

The total is the sum across cells, typically landing a useful corpus between 1,500 and 5,000 cases — small enough to source real truth for, large enough that no stratum is statistically empty.

Sizing Each Stratum

Stratum size is a statistics question with a practical answer. Too few rows and a stratum cannot detect a change smaller than its own noise; too many and the corpus becomes expensive to maintain and slow to run. The table below is the sizing we use for a merge gate.

Two points per stratum is the pragmatic gate. Below that, the run is fast and the result is noise; above it, every pull request waits on thousands of provider calls. Reserve the larger sizes for the scheduled run, where latency does not block anyone and the extra sensitivity is genuinely useful.

Row Schema

Each sampled address becomes one row. The schema below is the sampler’s output and the input to the corpus runner; keep the column names stable.

Store it as CSV for reviewability or Parquet at scale; a JSON array of the same fields works where a document store is more convenient. Whichever format, commit it under version control so a diff shows exactly which cases changed between releases.

The Sampler

The sampler takes a source frame that already carries the stratum columns and a per-stratum target, then draws that many rows from each cell with a seeded, reproducible groupby. Sampling is seeded so the same source frame and seed always yield the same test set — a non-reproducible corpus cannot anchor a regression baseline.

from __future__ import annotations

import hashlib
from typing import Mapping

import pandas as pd

# Default target per (country, address_type) cell; overridden by `targets`.
_DEFAULT_TARGET = 40


def _case_id(source_id: str, version: str) -> str:
    """Deterministic, collision-resistant id stable across reruns."""
    return hashlib.sha256(f"{source_id}|{version}".encode()).hexdigest()[:16]


def stratified_sample(
    frame: pd.DataFrame,
    version: str,
    targets: Mapping[tuple[str, str], int] | None = None,
    seed: int = 20260710,
) -> pd.DataFrame:
    """Draw a stratified ground-truth test set from a source address frame.

    Args:
        frame:    Source rows with at least source_id, raw_input, country,
                  region, address_type, truth_lat, truth_lon, truth_precision,
                  truth_source, truth_captured_date columns.
        version:  Corpus semver stamped onto every sampled row.
        targets:  Optional {(country, address_type): count} overrides.
        seed:     RNG seed for reproducible sampling.

    Returns:
        A DataFrame of sampled cases following the corpus row schema.
    """
    targets = targets or {}

    def _take(group: pd.DataFrame) -> pd.DataFrame:
        key = (group.name[0], group.name[1])
        n = min(targets.get(key, _DEFAULT_TARGET), len(group))
        return group.sample(n=n, random_state=seed)

    sampled = (
        frame.groupby(["country", "address_type"], group_keys=False)
        .apply(_take)
        .reset_index(drop=True)
    )

    sampled["case_id"] = sampled["source_id"].map(lambda s: _case_id(str(s), version))
    sampled["corpus_version"] = version
    # Negative rows carry no truth and expect a no-match outcome.
    is_negative = sampled["address_type"].eq("negative")
    sampled.loc[is_negative, ["truth_lat", "truth_lon"]] = pd.NA
    sampled["expected_outcome"] = is_negative.map({True: "no_match", False: "match"})

    cols = [
        "case_id", "raw_input", "country", "region", "address_type",
        "truth_lat", "truth_lon", "truth_precision", "truth_source",
        "truth_captured_date", "hit_threshold_m", "expected_outcome", "corpus_version",
    ]
    return sampled[cols]

The groupby(...).apply(_take) draws the target count from each stratum in one vectorized pass; min(target, len(group)) prevents an error when a thin cell has fewer rows than its target, which is itself a signal to source more truth for that cell.

Recording Provenance

Provenance is what lets a future reviewer decide whether a surprising error is a provider regression or a bad truth point. Every row must answer two questions: where did this coordinate come from, and when. The truth_source and truth_captured_date columns above carry exactly that. Prefer surveyed points where a field or telematics feed recorded the real location; fall back to an authoritative registry; use consensus (agreement among several independent geocoders within a tight radius) only when nothing better exists, and never to benchmark the same providers that produced it.

Provenance Makes the Corpus Auditable

A ground-truth row without provenance is an assertion. Six columns turn it into evidence: where the truth came from, when, who accepted it, and what the tolerance is. These are the columns that let a future disagreement be settled rather than argued.

The per-row tolerance is what lets one corpus cover both a dense city block and a remote farm without either a false failure or a missed regression. It also documents the intent: a row with a 500-metre tolerance is explicitly not testing rooftop precision, and nobody has to reverse-engineer that from the data.

Edge Cases

• Thin strata. If a cell holds fewer rows than its target, the sampler takes all of them and the corpus under-covers that class. Log the shortfall and treat it as a data-sourcing task, not a silent gap — the corpus only measures what it contains.
• Duplicate inputs across strata. The same address string can legitimately appear under different address_type labels (a unit and its parent street). Deduplicate on raw_input within a stratum but allow it across strata, since each tests a different resolution path.
• Truth in a national grid. Registry coordinates often arrive in a projected CRS. Reproject to WGS84 before writing truth_lat/truth_lon and assert the values fall within valid latitude and longitude ranges, or a whole stratum will show a uniform false error.

Keeping the Corpus Alive

A corpus decays. Addresses are demolished, renumbered and reassigned, and a test set that is never refreshed slowly accumulates rows that fail for reasons that have nothing to do with your pipeline. Three maintenance actions keep it honest.

The middle panel is the compounding one. Every confirmed regression that becomes a permanent row makes the gate strictly better at catching its own class of bug, and over a couple of years that accumulated set is worth more than the representative sample it sits beside.

Integration Note

This test set is the input the regression corpus workflow loads and scores on every build. Once assembled, the hit/miss classification each row receives depends on the confidence and outlier logic described in validating geocoding accuracy and confidence scoring, which determines whether a returned coordinate is trustworthy before its distance to truth is even measured.

---

Related

• Building regression corpora for geocoding accuracy — the workflow that runs this test set against providers and gates on the result.
• Measuring geocoding match-rate in Python — the metric functions that consume the schema defined here.
• Validating geocoding accuracy and confidence scoring — how each geocoded result is judged trustworthy before it is scored against truth.