Limiting Concurrency With Asyncio Semaphores

TL;DR: Size the semaphore from the rate limit multiplied by the median latency, hold it across the whole request including the response read, give each provider its own so a slow one cannot starve the others, and make sure the HTTP connector limit is at least as large or it becomes an invisible second bound. The surrounding dispatcher is described in building async geocoding requests in Python.

Sizing It From Little’s Law

Concurrency is not a free parameter. Given a rate limit and a latency, the number of in-flight requests needed to saturate the rate is fixed, and anything above it simply queues inside your own process.

Required concurrency = rate × latency Three provider configurations. At five requests per second with 200 millisecond latency, one concurrent request saturates the rate. At fifty per second, ten are needed. At three hundred per second, sixty are needed. A note explains that setting the semaphore far above the required figure only queues work inside the process. Permitted rate Median latency Concurrency needed 5 / s 200 ms 1 the limiter binds, not the semaphore 50 / s 200 ms 10 a comfortable working range 300 / s 200 ms 60 memory becomes the next constraint

Setting the semaphore to sixteen against a five-per-second provider is a common default and means fifteen tasks are permanently blocked on the rate limiter rather than on the network. That is harmless but misleading — it makes the semaphore look irrelevant, until latency rises and it suddenly becomes the binding constraint with no headroom.

Holding the Slot for the Right Span

from __future__ import annotations

import asyncio

import aiohttp


async def fetch_one(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    limiter,                      # shared rate limiter
    url: str,
) -> dict:
    """Acquire, request, read the body, release — in that order."""
    async with sem:                          # in-flight bound
        async with limiter:                  # start-rate bound
            async with session.get(url) as resp:
                resp.raise_for_status()
                return await resp.json()     # body read INSIDE the semaphore

The placement of await resp.json() is the detail that matters. Releasing the semaphore when the headers arrive and reading the body afterwards means the slot is free while the connection is still open and streaming, so the real in-flight count exceeds the bound — often by enough to exhaust the connection pool.

One Semaphore Per Provider

Shared slots versus per-provider bulkheads Two arrangements. With one shared semaphore of sixteen slots, a slow provider occupies fifteen of them and the healthy providers are starved. With a semaphore per provider, the slow one saturates its own eight slots while the others continue at full speed. one shared semaphore slow provider holds 15 of 16 slots healthy providers get one one degraded provider degrades the whole batch one semaphore per provider slow: 8/8 held healthy: 8/8 free the slowdown is confined to the records routed to that provider — everything else proceeds this is the bulkhead pattern, in one object

The per-provider arrangement also makes the numbers meaningful. A single global semaphore has to be sized for the slowest provider and is therefore wrong for the fastest; per-provider bounds can each be derived from that provider’s own latency, which is the only way the arithmetic above applies at all.

The Connector Limit Behind It

aiohttp applies its own cap through TCPConnector(limit=...), defaulting to one hundred total and, more importantly, limit_per_host defaulting to zero (unlimited) or a small number depending on version. Whichever is smaller — your semaphore or the connector — is the real bound, and only one of them is visible in your code.

connector = aiohttp.TCPConnector(
    limit=64,             # total across all hosts
    limit_per_host=32,    # must exceed the per-provider semaphore
    ttl_dns_cache=300,
    keepalive_timeout=60,
)
session = aiohttp.ClientSession(connector=connector, timeout=TIMEOUTS)

Set the connector generously and let the semaphores do the bounding. When the connector is the tighter of the two, tasks block inside the HTTP client rather than at your semaphore, which means your own instrumentation shows no waiting and the delay appears as latency — a confusing signal that sends people to investigate the provider.

Measuring Which Bound Is Binding

Instrumenting the two waits separately is a small change that answers a recurring question. Time spent waiting on the semaphore means the provider is slower than expected and requests are backing up; time spent waiting on the rate limiter means you are simply saturating the quota, which is the healthy state for a batch.

Emitting both as histograms, per provider, turns “the batch is slow” into one of two specific findings within seconds. It also makes tuning empirical: raising the semaphore only helps if semaphore wait is significant, and if limiter wait dominates then the only lever that matters is quota or a second provider.

Two Waits, Two Diagnoses

What each wait time tells you Two panels. Time spent waiting on the semaphore indicates the provider has slowed and requests are backing up, so the useful response is to investigate latency or add a fallback. Time spent waiting on the rate limiter indicates the quota is saturated, which is the healthy state for a batch and calls for more quota rather than more concurrency. semaphore_wait_seconds rising → the provider slowed down requests are backing up behind slow responses act on: latency, timeouts, a fallback provider raising the semaphore only hides it limiter_wait_seconds rising → the quota is saturated the healthy state for a batch at full speed act on: quota, a second provider, dedup concurrency changes nothing here One combined "slow" metric cannot distinguish these two, and they call for opposite responses.

Tuning by Observation, Not by Guess

The temptation with concurrency is to try a number, see whether the batch is faster, and keep it. That works until conditions change, and it leaves nobody able to explain the value.

A better loop takes ten minutes. Start from the computed figure — rate multiplied by median latency — and run a representative chunk. Record three things: total duration, mean semaphore wait and mean limiter wait. Then double the semaphore and repeat. If duration improves materially, the semaphore was binding and the new value is better; if it does not, the limiter is binding and the semaphore is already large enough.

Two iterations of that loop are usually sufficient to bracket the right value, and the recorded numbers are what justify it later. They also make the setting revisable: when provider latency changes, the same three numbers say immediately whether the old value is still right, without anyone having to reconstruct the original reasoning.

Finally, write the derivation down next to the constant. A comment saying “rate 50/s × median 200 ms ≈ 10, doubled for headroom” turns an arbitrary-looking 20 into a value the next person can check, and into one they will update rather than leave alone out of caution.

Edge Cases and Failure Modes

A semaphore shared across event loops. An asyncio.Semaphore binds to the loop that first awaits it. Creating one at module import and using it from a second loop — common in tests — raises or misbehaves; construct it inside the coroutine that owns the loop.

Releasing on the exception path. async with handles it; a manual acquire() followed by release() in the happy path only will leak a slot on every failure and the pipeline will grind to a halt after exactly n errors.

Semaphore larger than the pool. Tasks acquire a slot, then block on a connection, producing a queue that no metric attributes correctly. Keep the pool at or above the sum of the semaphores.

Unbounded task creation. A semaphore bounds concurrency, not memory: creating a million coroutine objects up front costs memory before any of them run. Iterate in chunks, as the batch throughput guide describes.

One structural point worth stating plainly: the semaphore belongs to the provider, not to the batch. Constructing it per batch means two concurrent batches in the same process each get their own, and the provider sees double the intended concurrency — the same multiplication problem that shared rate limiting exists to solve, reproduced one layer up. Create the semaphores once, alongside the client, and pass them down.

The same applies to the connector. A session per batch means a connection pool per batch, and the pool limits then multiply exactly as the semaphores do. One session, one connector and one set of semaphores per process, all created at start-up, is the arrangement where the numbers you configure are the numbers the provider experiences.

Integration Note

The semaphore is the bulkhead layer of the resilience stack — the outermost gate, deciding whether a call may start at all — and it composes directly with the distributed rate limiter, which bounds starts per second rather than concurrent starts. Holding both, per provider, is the arrangement that survives a provider slowing down without either over-running its quota or stalling the rest of the batch.

Where a process genuinely needs isolated pools — a low-latency interactive path beside a bulk batch path, say — make that separation explicit and size each one deliberately, rather than letting it emerge from where the objects happened to be constructed.

An explicit split also documents the intent for whoever reads the configuration next, which is worth more than the few lines it costs to write down.