Building a Stratified Ground-Truth Address Test Set

TL;DR: Partition a source frame of real addresses by country, region, and address type, draw a fixed target count from each cell with a pandas groupby().sample(), attach known-good coordinates plus a provenance tag to every row, and version the result. This assembles the stratified ground-truth set that the regression corpus workflow scores against.

A ground-truth test set is a table of addresses whose correct coordinates you already know, sampled so that every address class you care about is present in usable numbers. The quality of every downstream accuracy number depends on how this set is assembled, and the single decision that matters most is stratification: drawing a fixed quota from each meaningful cell rather than sampling proportionally from production, where the tail formats that break geocoders barely appear.

Choosing Strata

Stratify on the axes where geocoding quality genuinely changes. Three axes cover almost every pipeline:

• Country (ISO 3166-1 alpha-2). Postal conventions, provider coverage, and script all change hardest at national borders. This is nearly always the primary axis.
• Region (state, province, prefecture). Within a large country, coverage varies between dense metros and rural areas; a region axis stops one city dominating the sample.
• Address type (street, unit, po_box, rural_route, intersection, negative). Geocoders resolve each class through a different internal pipeline, so each needs its own cell.

The cross-product of these axes defines the strata. Keep only cells that correspond to a real quality boundary and that you can fill with truth data — an empty cell measures nothing. Add a negative address-type cell of inputs that should not resolve (malformed strings, fictional streets); it is the only way to catch a provider that starts confidently returning coordinates for garbage.

Target Count per Stratum

Assign a fixed target to each cell rather than a global fraction. A per-stratum match-rate needs enough cases to be stable: 30 is the practical floor for a rough rate, 50 or more gives a tighter interval. Weight the target up for strata that are both high-volume in production and high-risk.

The total is the sum across cells, typically landing a useful corpus between 1,500 and 5,000 cases — small enough to source real truth for, large enough that no stratum is statistically empty.

Row Schema

Each sampled address becomes one row. The schema below is the sampler’s output and the input to the corpus runner; keep the column names stable.

Store it as CSV for reviewability or Parquet at scale; a JSON array of the same fields works where a document store is more convenient. Whichever format, commit it under version control so a diff shows exactly which cases changed between releases.

The Sampler

The sampler takes a source frame that already carries the stratum columns and a per-stratum target, then draws that many rows from each cell with a seeded, reproducible groupby. Sampling is seeded so the same source frame and seed always yield the same test set — a non-reproducible corpus cannot anchor a regression baseline.

from __future__ import annotations

import hashlib
from typing import Mapping

import pandas as pd

# Default target per (country, address_type) cell; overridden by `targets`.
_DEFAULT_TARGET = 40


def _case_id(source_id: str, version: str) -> str:
    """Deterministic, collision-resistant id stable across reruns."""
    return hashlib.sha256(f"{source_id}|{version}".encode()).hexdigest()[:16]


def stratified_sample(
    frame: pd.DataFrame,
    version: str,
    targets: Mapping[tuple[str, str], int] | None = None,
    seed: int = 20260710,
) -> pd.DataFrame:
    """Draw a stratified ground-truth test set from a source address frame.

    Args:
        frame:    Source rows with at least source_id, raw_input, country,
                  region, address_type, truth_lat, truth_lon, truth_precision,
                  truth_source, truth_captured_date columns.
        version:  Corpus semver stamped onto every sampled row.
        targets:  Optional {(country, address_type): count} overrides.
        seed:     RNG seed for reproducible sampling.

    Returns:
        A DataFrame of sampled cases following the corpus row schema.
    """
    targets = targets or {}

    def _take(group: pd.DataFrame) -> pd.DataFrame:
        key = (group.name[0], group.name[1])
        n = min(targets.get(key, _DEFAULT_TARGET), len(group))
        return group.sample(n=n, random_state=seed)

    sampled = (
        frame.groupby(["country", "address_type"], group_keys=False)
        .apply(_take)
        .reset_index(drop=True)
    )

    sampled["case_id"] = sampled["source_id"].map(lambda s: _case_id(str(s), version))
    sampled["corpus_version"] = version
    # Negative rows carry no truth and expect a no-match outcome.
    is_negative = sampled["address_type"].eq("negative")
    sampled.loc[is_negative, ["truth_lat", "truth_lon"]] = pd.NA
    sampled["expected_outcome"] = is_negative.map({True: "no_match", False: "match"})

    cols = [
        "case_id", "raw_input", "country", "region", "address_type",
        "truth_lat", "truth_lon", "truth_precision", "truth_source",
        "truth_captured_date", "hit_threshold_m", "expected_outcome", "corpus_version",
    ]
    return sampled[cols]

The groupby(...).apply(_take) draws the target count from each stratum in one vectorized pass; min(target, len(group)) prevents an error when a thin cell has fewer rows than its target, which is itself a signal to source more truth for that cell.

Recording Provenance

Provenance is what lets a future reviewer decide whether a surprising error is a provider regression or a bad truth point. Every row must answer two questions: where did this coordinate come from, and when. The truth_source and truth_captured_date columns above carry exactly that. Prefer surveyed points where a field or telematics feed recorded the real location; fall back to an authoritative registry; use consensus (agreement among several independent geocoders within a tight radius) only when nothing better exists, and never to benchmark the same providers that produced it.

Edge Cases

• Thin strata. If a cell holds fewer rows than its target, the sampler takes all of them and the corpus under-covers that class. Log the shortfall and treat it as a data-sourcing task, not a silent gap — the corpus only measures what it contains.
• Duplicate inputs across strata. The same address string can legitimately appear under different address_type labels (a unit and its parent street). Deduplicate on raw_input within a stratum but allow it across strata, since each tests a different resolution path.
• Truth in a national grid. Registry coordinates often arrive in a projected CRS. Reproject to WGS84 before writing truth_lat/truth_lon and assert the values fall within valid latitude and longitude ranges, or a whole stratum will show a uniform false error.

Integration Note

This test set is the input the regression corpus workflow loads and scores on every build. Once assembled, the hit/miss classification each row receives depends on the confidence and outlier logic described in validating geocoding accuracy and confidence scoring, which determines whether a returned coordinate is trustworthy before its distance to truth is even measured.

---

Related

• Building regression corpora for geocoding accuracy — the workflow that runs this test set against providers and gates on the result.
• Measuring geocoding match-rate in Python — the metric functions that consume the schema defined here.
• Validating geocoding accuracy and confidence scoring — how each geocoded result is judged trustworthy before it is scored against truth.