Exporting Geocoding Pipeline Metrics to Prometheus

TL;DR: Use counters for events and histograms for durations, label only by provider, market and precision tier, push batch results to a gateway with a grouping key rather than exposing a scrape endpoint from a job that exits, and precompute the ratios as recording rules. The metric design itself is covered in monitoring and alerting for geocoding pipelines.

The Instrument Set

from prometheus_client import Counter, Histogram, Gauge

LOOKUPS = Counter(
    "geocode_lookups_total",
    "Address lookups attempted",
    ["provider", "market", "outcome"],       # outcome: hit|resolved|failed|skipped
)

RESOLVED_TIER = Counter(
    "geocode_resolved_tier_total",
    "Resolved lookups by precision tier",
    ["provider", "market", "tier"],          # tier: rooftop|parcel|street|locality
)

CALL_SECONDS = Histogram(
    "geocode_provider_call_seconds",
    "Provider call latency",
    ["provider"],
    buckets=(0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 8.0),
)

BILLED = Counter(
    "geocode_calls_billed_total",
    "Billable provider calls, including retries",
    ["provider"],
)

BREAKER = Gauge(
    "geocode_breaker_open",
    "1 when the circuit breaker for a provider is open",
    ["provider"],
)

Two choices there are deliberate. The bucket boundaries are geometric and centred on the range where geocoding latency actually lives, so the histogram has resolution where it matters rather than uniform resolution across an arbitrary range. And BILLED is separate from LOOKUPS because retries are billable but are not new lookups — collapsing them makes the cost ratio unusable.

Label Cardinality Is the Whole Design

Prometheus creates one time series per unique label combination, and a geocoding pipeline offers many tempting labels that would each multiply the series count by thousands.

Which labels are safe and what each costs Two panels. Safe labels are provider with about five values, market with about forty, tier with five and outcome with four, producing a manageable series count. Unsafe labels are address, cache key and customer identifier, each with millions of values, which would produce an unbounded series count and destabilise the metrics backend. safe labels provider ~5 values market ~40 values tier 5 values outcome 4 values worst case ≈ 4 000 series — comfortable never label by address millions cache_key millions customer_id millions trace_id unbounded these belong in logs, not in metrics Cardinality mistakes are not gradual: one deploy adding a high-cardinality label can take the backend down within an hour, and the series it created persist until retention expires.

The market label deserves a caveat: use a bounded set such as ISO country codes, and map anything unexpected to a literal other bucket. A label taken directly from input data is a cardinality bomb waiting for the first record with a malformed country field.

Instrument Choice, in One Table

Which instrument for which quantity Three rows. Counters measure monotonically increasing events such as lookups and billed calls, and are read through rate. Histograms measure durations such as provider latency and support percentile queries. Gauges measure instantaneous states such as breaker open, and must be set on every transition. Instrument Measures Mistake it prevents Counter lookups, billed calls, failures graphing a raw total that resets on deploy Histogram provider latency, batch duration averaging a distribution with a long tail Gauge breaker open, queue depth leaving a stale value that never clears

Batch Jobs Need a Push Gateway

A scrape-based collector cannot scrape a process that has already exited, which is the normal state of a nightly batch. Pushing the final values with a stable grouping key solves it, and the grouping key is what stops successive runs overwriting each other’s history in a way that hides failures.

from prometheus_client import CollectorRegistry, Counter, push_to_gateway

def report_batch(gateway: str, job_name: str, run_id: str, stats: dict[str, int]) -> None:
    """Push final batch counters. One registry per run, grouped by job."""
    registry = CollectorRegistry()
    rows = Counter(
        "geocode_batch_records_total", "Records processed by outcome",
        ["outcome"], registry=registry,
    )
    for outcome, count in stats.items():
        rows.labels(outcome=outcome).inc(count)

    push_to_gateway(
        gateway,
        job=job_name,
        registry=registry,
        grouping_key={"run_id": run_id, "instance": job_name},
    )

Push at the end of the run, not incrementally. A partially pushed batch looks like a completed small batch, and the resulting dip in volume is indistinguishable from a real one — which is exactly the confusion that makes people distrust batch dashboards.

Recording Rules for the Ratios

The metrics that matter are ratios, and computing them in every dashboard panel and alert is both slow and error-prone. Recording rules evaluate them once, server-side, and give everything downstream one definition to share.

groups:
  - name: geocoding
    interval: 60s
    rules:
      - record: geocode:match_rate:5m
        expr: |
          sum by (market) (rate(geocode_resolved_tier_total[5m]))
          /
          sum by (market) (rate(geocode_lookups_total[5m]))

      - record: geocode:rooftop_share:5m
        expr: |
          sum by (market) (rate(geocode_resolved_tier_total{tier="rooftop"}[5m]))
          /
          sum by (market) (rate(geocode_resolved_tier_total[5m]))

      - record: geocode:cache_hit_rate:5m
        expr: |
          sum(rate(geocode_lookups_total{outcome="hit"}[5m]))
          /
          sum(rate(geocode_lookups_total[5m]))

Naming them with the namespace:metric:window convention makes them recognisable as derived series, which matters when someone is exploring the metric browser and needs to know which values are raw and which are computed.

Naming That Survives Contact With Dashboards

Metric names are an interface, and renaming one breaks every dashboard, alert and saved query that referenced it. Three conventions keep them stable enough to live with.

Use a consistent prefix for everything the pipeline emits — geocode_ here — so a metric browser groups them together and a wildcard query can find them all. This sounds trivial until you inherit a system where the same pipeline emits geo_, geocoder_ and address_ prefixed series depending on which service wrote them.

Suffix by unit and type in the way the ecosystem expects: _total for counters, _seconds for durations, _bytes for sizes. The conventions are not enforced by the client, but every tool and every reader assumes them, and a duration named _ms will eventually be graphed as if it were seconds.

Avoid encoding in the name what should be a label. geocode_lookups_here_total and geocode_lookups_google_total cannot be summed with a single query, whereas one metric with a provider label can be summed, filtered and grouped freely. The rule of thumb is that anything you would ever want to aggregate across belongs in a label.

Finally, treat a rename as a migration rather than an edit: emit both names for a period, move the dashboards and alerts, then drop the old one. It costs a few weeks of duplicate series and avoids the alternative, which is discovering during an incident that the alert has been silently evaluating a metric that no longer exists.

Why Recording Rules Rather Than Dashboard Queries

One definition versus several Two arrangements. In the first, three dashboards and two alerts each carry their own copy of the match-rate expression, which drift apart over time. In the second, one recording rule computes it centrally and every consumer reads the same series. five copies of one expression three dashboards, two alerts, each edited separately they will disagree, and the argument costs more than the panels one recording rule evaluated once a minute, server-side every consumer reads the identical series dashboards alert rules The rule is also faster: the expression is evaluated once per interval rather than once per panel refresh.

Edge Cases and Failure Modes

Counters reset on restart. Prometheus handles this correctly through rate(), but a dashboard that graphs the raw counter will show a cliff at every deployment. Always graph rates, never raw counters.

Histograms with too many buckets. Each bucket is a series per label combination, so a twenty-bucket histogram labelled by provider and market is thousands of series on its own. Eight buckets is usually plenty for latency.

Gauges left stale. The breaker gauge must be set on both transitions, not just when it opens. A gauge that is only ever set to one stays at one forever, and the dashboard shows a permanent outage.

Push gateway retaining dead runs. Metrics pushed with a run identifier persist until deleted. Either delete the group at the end of a successful run, or use a stable grouping key without the run identifier and accept last-write-wins semantics.

Duplicate registration. Creating the same metric twice in one process raises at import time in some client versions and silently duplicates in others. Declare instruments at module scope, once, and import them rather than constructing them per call site.

One last operational point: keep the instrument definitions in a module that has no other responsibilities. A metrics module that imports the geocoding client creates an import cycle the first time the client wants to record something, and the usual workaround — importing inside the function — makes the hot path measurably slower. A leaf module with no internal imports avoids the problem entirely and makes the full metric surface readable in one file, which is worth more than it sounds when someone new asks what the pipeline exposes.

Integration Note

These instruments are the raw material for the alerts described in alerting on geocoding match rate drops, and the label set determines what those alerts can segment by — which is why the market label is worth carrying even when the pipeline serves one country today. The cost family depends on the same counters as the quota tracking in Redis; exporting both and reconciling them occasionally is a cheap way to catch an instrumentation gap in either.

The same module is the natural home for a short docstring describing what each metric means and when it moves, since that description is the thing every dashboard author and alert writer needs and nobody wants to reconstruct from the code.

Treat that docstring as part of the metric contract rather than as a comment.