Resilience Patterns for Geocoding APIs

As part of the Multi-API Routing & Fallback Chains architecture, resilience is the layer that decides whether a provider incident costs you a degraded hour or a failed batch. This page covers the five patterns that matter for geocoding specifically — timeouts, classified retries, circuit breakers, bulkheads and idempotency — and, more importantly, which failure each one actually answers.

The patterns are frequently deployed as a bundle without that mapping, which is how pipelines end up retrying a deterministic zero-result three times, or opening a breaker on a rate-limit response that simply needed a pause. Each pattern is cheap; applying the wrong one is not.

Prerequisites

One Pattern Per Failure

Each pattern answers exactly one question, and the value of naming them separately is that it becomes obvious when one is being asked to do another’s job.

Which resilience pattern answers which failure Five rows pairing a pattern with the failure it addresses. Timeouts answer a socket that never responds. Classified retries answer a transient error that a second attempt can fix. Circuit breakers answer sustained provider failure. Bulkheads answer one slow provider consuming shared resources. Idempotency keys answer duplicate work caused by the retries themselves. Pattern Answers Does not answer timeout a socket that never answers a provider that answers wrongly classified retry a transient error a deterministic zero result circuit breaker sustained provider failure a single unlucky request bulkhead one provider starving the rest exceeding a provider quota idempotency key duplicate work from retries the original failure

The right-hand column is the useful half of the table. A team that has added retries and still sees zero-result records failing has confused a deterministic answer with a transient one; a team whose breaker keeps opening on a healthy provider has confused a single slow request with sustained failure.

Step 1 — Bound Every Wait

An unbounded wait is the failure that turns a provider incident into a stalled batch. Most HTTP clients default to no total timeout, so a socket that accepts a connection and then goes quiet holds a worker indefinitely. Set three separate bounds, because they fail for different reasons.

import aiohttp

# Connect: how long to wait for a socket. Read: gap between bytes.
# Total: the hard ceiling on the whole request, including retries inside the client.
TIMEOUTS = aiohttp.ClientTimeout(
    total=8.0,          # nothing may take longer than this, ever
    connect=2.0,        # a healthy provider connects in well under a second
    sock_read=4.0,      # a stalled stream is dead long before this
)

session = aiohttp.ClientSession(timeout=TIMEOUTS)

Derive total from the provider’s own observed 99th percentile plus a margin, not from a round number. A ceiling well above the real tail lets a degraded provider hold workers for seconds each; a ceiling below the tail abandons responses that were about to arrive and converts them into billable retries.

Step 2 — Retry Only the Retryable

A retry is a bet that the same request will produce a different answer. That bet is good for a connection reset and bad for a well-formed address the provider simply does not know. Classify first, then decide, using the same taxonomy the fallback chain uses.

from dataclasses import dataclass


@dataclass(frozen=True)
class Decision:
    retry_same: bool
    next_provider: bool
    delay_s: float


def classify(status: int | None, exc: Exception | None) -> Decision:
    """Map a failure to a retry decision. One place, one table."""
    if exc is not None:                         # connection reset, timeout, DNS
        return Decision(retry_same=True, next_provider=False, delay_s=0.5)
    if status is None:
        return Decision(True, False, 0.5)
    if status == 429:                           # throttled: wait, do not escalate
        return Decision(True, False, 2.0)
    if 500 <= status < 600:                     # transient server-side
        return Decision(True, False, 1.0)
    if status in (400, 404, 422):               # deterministic: escalate instead
        return Decision(False, True, 0.0)
    if status in (401, 402, 403):               # auth or quota: stop using this one
        return Decision(False, True, 0.0)
    return Decision(False, True, 0.0)

Keeping this as a pure function of status and exception makes it testable without a network and gives the whole pipeline one place where the policy lives. When a provider starts returning a novel status code, exactly one function changes.

Step 3 — Break the Circuit, Then Probe

Retries handle a bad second; a breaker handles a bad ten minutes. The state machine is covered in detail on the section overview, and the operational point here is where the state lives: shared, not per process.

Per-process versus shared breaker state Two panels. With per-process state, each of twenty workers independently makes its threshold number of failing calls before opening its own breaker, multiplying the doomed traffic by twenty. With shared state in Redis, the first worker to cross the threshold opens the breaker for all of them, so the doomed traffic is bounded by the threshold itself. per-process state 20 workers × 5-failure threshold = 100 doomed calls before the last worker stops trying and 20 independent probes on recovery each of them billable shared state one counter in Redis, one threshold = 5 doomed calls total, then every worker sees the open breaker and exactly one probe on recovery a lock decides which worker probes The recovery probe needs the lock as much as the threshold does — twenty simultaneous probes are a small stampede.

The probe lock is the part most implementations omit. When the cool-down elapses, every worker sees a half-open breaker and sends its own probe, which is precisely the burst the breaker existed to prevent. A short-lived Redis lock around the probe reduces that to one request.

Step 4 — Isolate With Bulkheads

Concurrency limits are usually applied globally, which quietly couples providers together: a slow provider holds connections while waiting, and healthy providers queue behind it. A per-provider semaphore — a bulkhead — bounds that blast radius.

Resource Shared pool Per-provider bulkhead
Connections one slow provider can hold all of them each provider capped independently
Worker slots a stall in one blocks every lookup a stall degrades only its own share
Failure blast radius whole pipeline one branch of the chain
Tuning one number, wrong for everyone one number per provider, from its own latency

The cost is a little more configuration and one extra semaphore per provider. The benefit is that a provider incident degrades the records routed to that provider instead of the entire batch, which is the difference between a partial result and no result.

Step 5 — Make Duplicates Cheap

Every retry risks doing the work twice: the first attempt may have succeeded and lost its response on the way back. For a metered read that means paying twice for one answer. A stable per-lookup key fixes the accounting on both sides.

import hashlib


def lookup_key(canonical_address: str, provider: str, attempt_group: str) -> str:
    """Stable identifier for one logical lookup, reused across retries."""
    raw = f"{provider}|{attempt_group}|{canonical_address}".encode()
    return hashlib.blake2b(raw, digest_size=16).hexdigest()

attempt_group is a batch or job identifier, not an attempt counter — the key must stay the same across retries of the same logical lookup and differ between genuinely separate requests for the same address. Send it as the provider’s idempotency header where one exists, and use it as the cache key regardless, so a retried lookup that already succeeded is answered locally rather than re-billed.

Composing the Five

Order matters when these are stacked. The bulkhead is outermost — it decides whether the call may start at all — then the breaker, then the retry loop, with the timeout innermost bounding each individual attempt.

How the five patterns nest around one call Four nested frames around a single HTTP attempt. The outermost is the per-provider bulkhead controlling admission. Inside it is the circuit breaker, which rejects immediately when open. Inside that is the retry loop with classified delays. Innermost is the timeout that bounds one attempt. The idempotency key travels with every attempt. bulkhead — may this call start? breaker — is this provider worth calling? retry loop — classified, jittered, capped at 3 timeout — bounds one attempt idempotency key travels with every attempt inside this frame Inverting any two layers breaks something: retries outside the breaker defeat it, and a timeout outside the retry loop bounds the wrong thing.

The inversion to watch for is putting the retry loop outside the breaker. It looks equivalent and is not: the breaker then sees one failure per logical lookup instead of one per attempt, so it takes three times as long to open, and the retries it was supposed to suppress happen anyway.

What to Measure

Resilience settings are guesses until they are measured, and four metrics turn them into evidence. Emit all four per provider, because the whole point of the chain is that providers behave differently and the numbers that matter are the ones that differ.

Attempt count per successful lookup is the first. In a healthy pipeline it sits just above one; a value climbing toward two means either the retry classifier is retrying things it should not, or a provider is genuinely degrading. The metric is cheap to emit and is the earliest signal of both.

Timeout rate is the second, and it must be split by which bound fired. Connect timeouts point at the network or a saturated pool; read timeouts point at the provider stalling mid-response; total timeouts point at a provider that is merely slow. Three different causes, three different fixes, one useless number if they are aggregated.

Breaker state duration is the third: how long each provider spent open, half-open and closed over the day. A breaker that never opens is either well-tuned or not wired in, and the only way to tell the two apart is to look at the failure counter that feeds it. A breaker that spends hours open is telling you the fallback chain, not the breaker, is carrying the pipeline.

The fourth is the deadline-exceeded rate — the share of logical lookups abandoned because the overall deadline elapsed while the retries and fallbacks were still in progress. This is the number that connects resilience settings to batch duration, and it is the one to watch when a nightly job starts finishing late without any obvious error.

Track all four as rates rather than counts. Counts rise with traffic and hide behaviour changes; rates stay flat while the system is healthy, which makes any movement worth reading.

None of these four needs a dashboard to be useful. A weekly digest naming the provider with the highest attempt count and the provider that spent the longest with an open breaker is enough to keep the settings honest, and it takes far less effort to maintain than a set of panels nobody opens between incidents.

Troubleshooting

The breaker opens during normal operation. The failure window is counted in requests rather than time, so a quiet period never ages out old failures. Switch to a time window.

Latency rises but no timeouts fire. The total timeout is set well above the real tail — common when it was copied from another service. Recompute it from this provider’s percentiles.

Retries make an incident worse. Either the delay has no jitter, so every worker retries in lockstep, or a deterministic class is being retried. Both are visible in the attempt-count metric broken down by status code.

Records fail while a provider is healthy. A bulkhead sized too small queues requests until they exceed the total timeout. Bulkhead capacity must exceed the concurrency the rate limiter will actually permit, or the two fight each other.

FAQ

How many retries should a geocoding call get?

Two or three, and only for transient classes. Beyond three the marginal success probability is small and the cost is certain, because every attempt is billable. A record that fails three transient attempts is better served by the next provider in the chain than by a fourth attempt at the same one.

Should the timeout be the same for every provider?

No. Derive it from each provider’s own observed latency distribution — roughly the 99th percentile plus a margin. A timeout set from a fast provider’s profile will abandon a slower one’s healthy responses, and a timeout set from the slowest will let a fast provider hang for seconds before anything notices.

Where should the circuit breaker state live?

In shared storage when several workers call the same provider, typically Redis. Per-process breakers each learn the provider is down independently, so with twenty workers you make twenty times the doomed calls before any of them stops.

Do geocoding requests need idempotency keys if they are read-only?

They are read-only to you and metered by the provider, so a duplicate is billed twice even though it changes nothing. Where a provider supports an idempotency or request-id header, sending a stable per-lookup key lets it collapse duplicates; where it does not, the same key still deduplicates in your own cache and logs.

What is the difference between a bulkhead and a rate limiter here?

A rate limiter bounds how fast you call one provider; a bulkhead bounds how much of your own resource any single provider can consume. A slow provider without a bulkhead will occupy every connection in a shared pool while waiting, starving the healthy providers even though none of them is over its rate.