Implementing a Circuit Breaker for Geocoding Providers

TL;DR: Count failures in a time window, keep the state in Redis so every worker shares one view, and put a short lock around the half-open probe so recovery is tested by one request rather than by the whole fleet. The pattern’s place in the wider stack is described in resilience patterns for geocoding APIs.

The State the Breaker Needs

Three values per provider are enough: a rolling record of recent failures, the instant the breaker may next be probed, and a counter of consecutive failed probes for the backoff. All three live in Redis so that every worker reads the same view.

Splitting the decision key from the counting key matters for cost. Every request consults the breaker, and a design that recounts the failure window on each call turns a microsecond decision into a range query. Reading a single timestamp keeps the hot path to one round trip.

The Implementation

from __future__ import annotations

import time
from dataclasses import dataclass

import redis


@dataclass(frozen=True)
class BreakerConfig:
    window_s: float = 60.0        # failures older than this no longer count
    threshold: int = 5            # failures inside the window that open it
    base_cooldown_s: float = 30.0
    max_cooldown_s: float = 600.0


class ProviderBreaker:
    """Shared-state circuit breaker keyed by provider name."""

    def __init__(self, r: redis.Redis, provider: str, cfg: BreakerConfig) -> None:
        self._r = r
        self._cfg = cfg
        self._fails = f"cb:{provider}:fails"
        self._open_until = f"cb:{provider}:open_until"
        self._probe_fails = f"cb:{provider}:probe_fails"
        self._probe_lock = f"cb:{provider}:probe_lock"

    def allows(self) -> bool:
        """True when a call may proceed — the only check on the hot path."""
        raw = self._r.get(self._open_until)
        if raw is None:
            return True                       # closed
        if time.time() < float(raw):
            return False                      # open, still cooling down
        # Cool-down elapsed: exactly one worker gets the probe.
        return bool(self._r.set(self._probe_lock, "1", nx=True, ex=10))

    def record_success(self) -> None:
        pipe = self._r.pipeline()
        pipe.delete(self._fails, self._open_until, self._probe_fails, self._probe_lock)
        pipe.execute()

    def record_failure(self) -> None:
        now = time.time()
        pipe = self._r.pipeline()
        pipe.zadd(self._fails, {f"{now}:{id(self)}": now})
        pipe.zremrangebyscore(self._fails, 0, now - self._cfg.window_s)
        pipe.zcard(self._fails)
        pipe.expire(self._fails, int(self._cfg.window_s * 2))
        count = pipe.execute()[2]

        if count >= self._cfg.threshold:
            attempts = self._r.incr(self._probe_fails)
            cooldown = min(
                self._cfg.base_cooldown_s * (2 ** (attempts - 1)),
                self._cfg.max_cooldown_s,
            )
            self._r.set(self._open_until, now + cooldown)
            self._r.delete(self._probe_lock)

The nx=True on the probe lock is the whole recovery mechanism. When the cool-down elapses, every worker reaches the same branch simultaneously, and exactly one wins the lock; the rest see the breaker as still open and route to the next provider. Without it, recovery begins with a burst equal to your worker count.

Tuning the Two Numbers

Threshold and window together define what “failing” means, and the two most common mistakes pull in opposite directions.

The window is the setting people get wrong most often, usually by counting attempts instead of seconds. A count-based window has no notion of time, so five failures spread across an hour trip the breaker exactly as readily as five failures in a second — and the first is normal operation while the second is an outage.

Why the Window Must Be Time-Based

The upper timeline is what a large batch looks like on an ordinary day: a handful of transient failures across many thousands of successful calls. Opening the breaker there takes a healthy provider out of service and pushes traffic onto a fallback that is usually worse, more expensive, or both.

Reading the Breaker’s Behaviour

Two metrics tell you whether the settings are right, and both are trivial to emit from the methods above. The first is time-in-state per provider per day; the second is the count of probes and their outcomes.

A breaker that never opens is either well tuned or not connected — check that record_failure is actually reached on the error paths, because it is easy to add the breaker to the happy path and forget the branch where a timeout is raised. A breaker that opens several times an hour and closes immediately is oscillating, which usually means the threshold is below the provider’s ordinary noise floor.

The probe outcome series is the more interesting one during an incident. A run of failed probes with a growing cool-down is the system behaving exactly as designed; a run of successful probes followed by immediate re-opening means the provider is flapping, and the right response is to raise the base cool-down so each recovery attempt gets more room.

Testing a Breaker Without a Provider

A breaker is easy to test badly, because the obvious test — point it at a broken provider and watch — is slow, flaky and dependent on someone else’s outage. Test it against a fake clock and a fake failure source instead, and every branch becomes deterministic.

Inject the time function rather than calling time.time() directly. With an injectable clock, “the cool-down elapses” is an assignment rather than a sleep, and a test that exercises the full open, half-open and closed cycle runs in milliseconds. The same seam lets you assert the geometric growth precisely instead of approximately.

Use a real Redis instance rather than a mock. The breaker’s correctness depends on Redis semantics — the atomicity of SET NX, the behaviour of sorted-set range removal — and a mock that implements those approximately will happily pass tests for code that races in production. A container running for the duration of the suite is cheap and tests the thing you actually deploy.

Write one test per transition and one for the concurrency property. The transitions are straightforward; the concurrency test — spawn twenty coroutines against a breaker whose cool-down has just elapsed, assert exactly one obtains the probe lock — is the one that catches the mistake this design exists to prevent, and it is the test most often missing.

Cool-Down Growth Over an Outage

Edge Cases and Failure Modes

Redis itself becomes unavailable. The breaker must fail open, not closed: if the state store cannot be read, allow the call. A breaker that blocks all traffic because its bookkeeping is down converts a Redis incident into a total outage.

Clock skew between workers. The open-until value is an absolute timestamp, so workers whose clocks differ by seconds will disagree about when the cool-down ended. Keep hosts on NTP, and prefer a generous cool-down over a tight one where skew is possible.

A provider that fails only for some inputs. A breaker is a per-provider signal and cannot express “this provider fails on Japanese addresses”. Those cases belong in the routing registry, not the breaker, or the breaker will open on a provider that is healthy for the traffic actually routed to it.

Probe lock outliving its usefulness. The lock carries a short expiry so a worker that dies mid-probe does not block recovery forever. Ten seconds is comfortably longer than one request and short enough that a crash costs one cool-down at most.

Integration Note

The breaker sits between the bulkhead and the retry loop described in resilience patterns for geocoding APIs, and its open state is what the fallback chain reads when it skips a provider without calling it. Stamping the skip reason onto the record — breaker-open rather than provider-error — is what later lets you distinguish an incident from a coverage gap when reviewing why a batch fell back so often.

Related

• Resilience Patterns for Geocoding APIs — where the breaker sits among the other four patterns.
• Retry With Exponential Backoff and Jitter in Python — the loop the breaker wraps.
• Implementing Fallback Chains for Failed Lookups — what happens to a record when the breaker says no.