Scoring Provider Results to Pick a Winner

TL;DR: Rank on the normalised precision tier first, break ties on component-match completeness, then on geometric plausibility, and finally on a stable provider preference so the outcome is deterministic. Keep the runner-up and the margin — a close call between two providers is the most useful review sample you have. The chain itself is described in implementing fallback chains for failed lookups.

First-Answer-Wins Is a Policy, Not a Default

The simplest chain accepts whatever the first responding provider returns. That is defensible when the chain is ordered by quality and the fallback is genuinely worse, and it is wrong as soon as the fallback sometimes returns a better answer — which is normal, because coverage varies by street, not by provider.

What first-answer-wins can throw away Two policies applied to the same record. Under first-answer-wins, the primary provider returns a street-level result and the chain stops, discarding a rooftop match the fallback would have supplied. Under scoring, both are obtained and the rooftop result wins. first answer wins provider A → street level · accepted provider B → never called cheapest, and silently accepts a weak result correct when the chain order really is a quality order score and choose provider A → street level provider B → rooftop · wins costs a second call for records below a tier floor apply it selectively, not to every record

The practical compromise is a tier floor: accept the first answer when it reaches rooftop or parcel precision, and consult the next provider only when it does not. That confines the extra spend to the records where a better answer plausibly exists, which on most corpora is a small minority.

The Comparable Score

from __future__ import annotations

from dataclasses import dataclass

TIER_RANK = {"rooftop": 4, "parcel": 3, "street": 2, "locality": 1, "postcode": 0}
PROVIDER_PREFERENCE = {"here": 2, "google": 1, "osm": 0}     # stable tie-break


@dataclass(frozen=True)
class Candidate:
    provider: str
    tier: str
    components_confirmed: int      # how many of street, number, unit, postcode matched
    inside_expected_area: bool     # geometric plausibility check
    lat: float
    lon: float


def score(c: Candidate) -> tuple[int, int, int, int]:
    """Lexicographic score — higher is better, and fully deterministic."""
    return (
        TIER_RANK.get(c.tier, -1),
        c.components_confirmed,
        1 if c.inside_expected_area else 0,
        PROVIDER_PREFERENCE.get(c.provider, -1),
    )


def pick(candidates: list[Candidate]) -> tuple[Candidate, Candidate | None, int]:
    """Return (winner, runner_up, tier_margin)."""
    ranked = sorted(candidates, key=score, reverse=True)
    winner = ranked[0]
    runner_up = ranked[1] if len(ranked) > 1 else None
    margin = (
        score(winner)[0] - score(runner_up)[0] if runner_up is not None else 0
    )
    return winner, runner_up, margin

A lexicographic tuple rather than a weighted sum is deliberate here. Weights invite a result that is worse on the dimension that matters most but wins on aggregate, and the ordering above states plainly that precision beats completeness beats plausibility beats habit.

Why the Provider Preference Is Last and Still Necessary

Deterministic tie-breaking Two candidates from different providers are identical on precision tier, component match and geometric plausibility. Without a stable final key, the winner depends on iteration order and can change between runs, producing coordinates that appear to move for no reason. candidate A · here tier rooftop · 4 components · inside area score (4, 4, 1, 2) wins on the final key candidate B · google tier rooftop · 4 components · inside area score (4, 4, 1, 1) identical on everything that matters Without a stable final key the winner depends on iteration order, so the stored coordinate changes between runs and every diff-based check reports spurious movement.

Determinism matters more than the preference itself. Two providers that genuinely agree will place a building a few metres apart, and a pipeline that alternates between them produces a stream of tiny coordinate changes that pollute every change-detection process downstream.

Keep the Runner-Up

Storing the losing candidate costs one nested object per record and buys three things that are hard to obtain otherwise. First, an audit trail: when a coordinate is questioned, the alternative that was considered is right there. Second, a disagreement metric: the distance between winner and runner-up is a free, per-record quality signal that needs no ground truth. Third, a review sample — the records where two independent corpora disagree by more than a hundred metres are the highest-yield use of a limited manual review budget.

The margin is worth storing separately from the runner-up itself. A margin of zero on the tier means the choice came down to a tie-break, and a batch where that share is rising is telling you the providers have converged and the chain may be doing less than it costs.

Disagreement Distance as a Review Queue

Where two providers disagree, and by how much A distribution of the distance between the winning and runner-up coordinates for the same address. Most records agree within a few metres. A small tail beyond one hundred and fifty metres is marked as the highest-yield sample for manual review. > 150 m — review these 0 m 1 km+ This signal needs no ground truth, which is what makes it the cheapest quality metric in the whole pipeline.

When to Consult a Second Provider at All

Scoring only matters when more than one candidate exists, and obtaining that second candidate costs money. The policy that decides when to spend it is as important as the scoring itself.

A tier floor is the usual rule: if the first answer reaches rooftop or parcel precision, stop; otherwise ask the next provider. On a typical corpus that triggers a second call on somewhere between five and fifteen percent of records, which is affordable and targets the spend precisely at the records where improvement is possible.

Two refinements are worth adding. First, make the floor per market: in a market where the primary provider rarely reaches rooftop, a rooftop floor sends nearly everything to the second provider and the cost is no longer marginal. Second, respect the cache — a second opinion obtained once should be stored with the record so the policy does not re-trigger on every subsequent run for the same address.

There is also a case for consulting a second provider on a small random sample of high-confidence records, independent of the floor. Those samples are what tell you whether the primary’s confident answers are actually right, and without them a chain can drift into trusting a provider that has quietly degraded — the failure the false-confident rate is designed to expose.

Edge Cases and Failure Modes

Comparing raw provider confidence. A 0.9 from one provider is not a 0.9 from another, which is why the score starts from a normalised tier rather than from the native field. The normalisation itself is covered in validating geocoding accuracy and confidence scoring.

Scoring results obtained with different parameters. If one provider received a country hint and another did not, the comparison measures the hint, not the provider. Send identical parameters or record the difference.

A plausibility check with missing polygons. inside_expected_area must be tri-state in practice — true, false, or unknown — and an unknown should not be scored as a failure, or every record in a market without boundary data is penalised.

Selective scoring that becomes universal. A tier floor set too high sends every record to a second provider and doubles the bill. Watch the share of records that trigger the second call; it should be a minority and stable.

A last practical note on implementation: keep the score function pure and free of I/O. It takes candidates and returns an ordering, which makes it trivially testable with fabricated inputs and keeps the interesting cases — ties, missing plausibility data, an unknown tier — covered by fast unit tests rather than by integration runs against live providers.

That purity also makes the function safe to reuse offline. Replaying historical candidate pairs through a modified scoring rule is how you evaluate a change to the ordering before shipping it, and that replay is only possible if the function needs nothing but the data already stored on the record.

Integration Note

Scoring sits at the end of the fallback chain, after the reconciliation step that maps provider-specific match types onto the shared tier scale. The winner’s tier is what the cache TTL policy reads, and the same tier ranking is what the background revalidation guard uses to ensure an entry is never replaced by a worse one — three consumers, one ordering, which is the argument for defining it in exactly one module.

Storing the candidate list, rather than only the winner, is what makes that replay possible at all — which is the practical reason the runner-up column earns its space in the schema alongside the audit and review-sampling arguments above.

Keeping both candidates also means a scoring change can be evaluated against months of real disagreements rather than against a handful of fabricated ones, which is the difference between a tuned rule and a guessed one.

Six months of stored candidate pairs is a dataset most teams never realise they could have had, and it costs one nested column to start collecting today.