Generating Canonical Address Keys in Python

TL;DR: A canonical address key is a deterministic hash of a normalized string — lowercase, NFKC-fold, strip punctuation, expand USPS abbreviations, drop unit noise, sort tokens, then blake2b. Two spellings of the same location produce the same key. This page supports deduplicating addresses by fuzzy canonical key, where that key does the cheap bulk of duplicate collapsing before any fuzzy scoring runs.

What Makes a Key “Canonical”

A canonical key must satisfy two properties: determinism (the same input always yields the same key, on every machine and every run) and equivalence collapse (all spellings that denote the same location yield the same key). The first rules out anything order-dependent — Python set iteration, dict ordering across versions, or locale-sensitive casing. The second is what the transform pipeline delivers: each step erases one axis of spurious variance while preserving the information that actually distinguishes addresses.

The hash at the end is not for security; it is for producing a fixed-width, index-friendly, collision-resistant token that is cheap to GROUP BY and to store as a foreign key. blake2b is chosen over md5/sha1 because it is fast, has no deprecation baggage, and lets you set the digest size directly.

One Address, Six Spellings, One Key

The point of canonicalisation is easiest to see on real variants. Every row below is the same delivery point as typed by a different system or person, and every one must reduce to an identical key — otherwise each becomes a separate cache entry, a separate provider call and a separate customer record.

Six input spellings reducing to one key Six input strings on the left, each a different spelling of 100 North Main Street Suite 5 — varying in case, punctuation, spelled-out versus abbreviated suffix and directional, and unit designator. All six converge through a transform stage into the single canonical key one hundred n main st ste 5. as received 100 North Main Street, Suite 5 100 N. Main St. Ste 5 100 N MAIN ST STE 5 100 n main street #5 100 N Main Str., Suite #5 100 n main st unit 5 canonicalise fold · strip · abbreviate squeeze · order 100 n main st ste 5 Row four is the one that catches naive implementations: a hash symbol as a unit designator, and doubled internal spaces that survive a strip() but not a whitespace squeeze.

Notice what the key does not keep: the recipient name, the country when it is implied, and any punctuation. Each of those varies between systems for reasons that have nothing to do with the delivery point, and every one you retain splits records that should have merged.

Transform Table

Each step targets one specific source of variance. Order is load-bearing: folding Unicode before stripping punctuation prevents accented letters from being discarded, and expanding abbreviations before sorting keeps st and street from sorting to different slots.

# Transform Removes variance in “123 N. Main St., Apt 4” becomes
1 lower() Letter case 123 n. main st., apt 4
2 NFKC normalize Unicode composition, full-width forms (code points folded)
3 Strip combining marks Accents / diacritics 123 n. main st., apt 4
4 Drop unit noise Suite / apartment / floor 123 n. main st.,
5 Strip punctuation ., ,, # 123 n main st
6 Collapse whitespace Repeated spaces 123 n main st
7 Expand abbreviations Suffix / directional spelling 123 north main street
8 Sort tokens Token order (unit before/after street) 123 main north street
9 blake2b hash — (produces the key) a1f3…

What Belongs in the Key and What Does Not

Every field you include splits records that differ on it; every field you exclude merges records that differ on it. The line is drawn by asking whether two records differing only on that field are the same delivery point — and the answer is not the same for every consumer, which is why unit designators sit on the boundary.

Included, excluded and borderline key fields Three panels. Always included are the civic number, normalised street name, suffix, locality and postcode, which together identify a delivery point. Never included are the recipient name, punctuation, casing and free-text notes, which vary between systems for reasons unrelated to location. Consumer-dependent are the secondary unit and building name, which should be included for parcel delivery and excluded for site-level analytics. always in civic number street name + suffix locality · postcode these identify the delivery point itself never in recipient name punctuation · casing delivery notes these vary between systems, not locations depends on the reader secondary unit building name in for parcel delivery, out for site analytics emit both keys

Emitting two keys — one with the unit and one without — costs one extra column and settles the argument permanently. Parcel routing joins on the unit-bearing key; a building-level analytics job joins on the other; and neither consumer has to compromise on a definition that was never going to fit both.

The Key Builder

The implementation compiles every regex once at module level, carries explicit type hints, and validates input before doing any work. The USPS map here is abbreviated; a production version loads the full Publication 28 suffix and directional set. See the official USPS Publication 28 for the authoritative abbreviation list.

from __future__ import annotations

import hashlib
import re
import unicodedata
from typing import Dict, Final

# Compile all patterns once at import time — never inside the hot function.
_PUNCT: Final = re.compile(r"[^\w\s]")
_WS: Final = re.compile(r"\s+")
_UNIT: Final = re.compile(
    r"\b(?:apt|apartment|ste|suite|unit|fl|floor|rm|room|bldg|building)\b.*$"
)

# USPS Publication 28 suffixes + directionals (trimmed for illustration).
_ABBREV: Final[Dict[str, str]] = {
    "st": "street", "str": "street", "ave": "avenue", "av": "avenue",
    "blvd": "boulevard", "rd": "road", "dr": "drive", "ln": "lane",
    "ct": "court", "cir": "circle", "pl": "place", "ter": "terrace",
    "hwy": "highway", "pkwy": "parkway", "sq": "square", "trl": "trail",
    "n": "north", "s": "south", "e": "east", "w": "west",
    "ne": "northeast", "nw": "northwest", "se": "southeast", "sw": "southwest",
}


def canonical_string(raw: str, drop_unit: bool = True) -> str:
    """Return a deterministic canonical form of a single-line address.

    Args:
        raw: The raw address string.
        drop_unit: If True, remove secondary-unit designators (per-building grain).

    Returns:
        A normalized, abbreviation-expanded, token-sorted string.

    Raises:
        TypeError: If raw is not a string.
    """
    if not isinstance(raw, str):
        raise TypeError(f"expected str, got {type(raw).__name__}")
    if not raw.strip():
        return ""

    # 1-3: case, Unicode compatibility fold, strip accents.
    text = unicodedata.normalize("NFKC", raw).lower()
    text = "".join(
        c for c in unicodedata.normalize("NFKD", text)
        if not unicodedata.combining(c)
    )
    # 4: drop secondary-unit noise before punctuation removal.
    if drop_unit:
        text = _UNIT.sub("", text)
    # 5-6: strip punctuation, collapse whitespace.
    text = _WS.sub(" ", _PUNCT.sub(" ", text)).strip()
    # 7: expand abbreviations token by token.
    tokens = [_ABBREV.get(tok, tok) for tok in text.split()]
    # 8: deterministic token order.
    tokens.sort()
    return " ".join(tokens)


def canonical_key(raw: str, drop_unit: bool = True) -> str:
    """Hash the canonical string into a stable 16-byte hex key.

    Empty or whitespace-only input yields a fixed sentinel key so that
    blank records cluster together instead of raising.
    """
    canon = canonical_string(raw, drop_unit=drop_unit)
    return hashlib.blake2b(canon.encode("utf-8"), digest_size=16).hexdigest()

Note that the unit regex is applied before punctuation stripping so that Apt 4, is removed cleanly with its trailing comma. Reordering these two steps would leave a dangling comma token that pollutes the key.

Vectorized pandas Example

For batch tables, avoid a per-row Python call where possible, but since the transform is string-heavy the practical pattern is a single .map over the column, then a vectorized hash. Compute the canonical string once and reuse it — it feeds both the key and any downstream rapidfuzz scoring.

import hashlib

import pandas as pd


def add_canonical_key(df: pd.DataFrame, addr_col: str) -> pd.DataFrame:
    """Attach canonical string and canonical key columns to a DataFrame.

    Args:
        df: Input frame containing an address column.
        addr_col: Name of the raw address column.

    Returns:
        A copy with '_canon' and '_key' columns added.
    """
    out = df.copy()
    out["_canon"] = out[addr_col].fillna("").astype(str).map(canonical_string)
    out["_key"] = out["_canon"].map(
        lambda c: hashlib.blake2b(c.encode("utf-8"), digest_size=16).hexdigest()
    )
    return out


# Deduplicate: one survivor per canonical key.
def collapse(df: pd.DataFrame, addr_col: str) -> pd.DataFrame:
    """Collapse exact canonical-key duplicates, keeping the first row per key."""
    keyed = add_canonical_key(df, addr_col)
    return keyed.drop_duplicates(subset="_key", keep="first")

Version the Key, Because It Will Change

The canonicalisation rules are code, and code changes. When they do, every existing key becomes unreachable — the cache misses everywhere, and any table joined on the key silently stops matching. Carrying an explicit version turns that from an incident into a planned migration.

Migrating a key derivation without a cache flush A three-phase migration. In phase one, only version six keys exist. In phase two, writes produce version seven keys while reads fall back to version six, and a backfill job rewrites stored rows. In phase three, version six is dropped once backfill coverage is complete. before key_v6 only steady state migration window write v7, read v7 then v6 backfill rewrites stored rows after key_v7 only v6 read path removed The read-through-to-v6 step is what keeps the provider bill flat during the window: a v7 miss finds the v6 entry, rewrites it under the new key and returns it, so no address is ever re-resolved just because the rules changed. Track backfill coverage as a percentage and only remove the fallback when it reaches 100.

Treat the version as a first-class artefact: a constant in one module, stamped into every key and every stored row, and asserted in tests. The failure this prevents — two services deriving keys with different rules and neither noticing — is invisible in every metric except a slowly rising duplicate rate.

Edge Cases

Empty and Non-String Input

Real tables carry nulls, integers coerced from spreadsheets, and whitespace-only cells. canonical_string raises TypeError on non-strings so silent coercion never corrupts a key, and returns "" for blank input so blank rows share a single sentinel key rather than each hashing to a different value. In pandas, fillna("").astype(str) normalizes the column before mapping.

Grain Determines Unit Handling

With drop_unit=True, 123 Main St Apt 4 and 123 Main St Apt 9 produce the same key (per-building grain). With drop_unit=False, the unit token survives, is abbreviation-normalized, and discriminates the two. Choose per pipeline and keep it consistent; mixing grains within one key column produces silent over- or under-merging. The trade-off is discussed in deduplicating addresses by fuzzy canonical key.

Versioning the Key Function

Any change to the abbreviation map or transform order changes the output key for some inputs, invalidating every previously stored key. Treat the function as versioned: embed a version tag alongside the stored key (for example v2:a1f3…) and recompute in a controlled backfill when the logic changes, exactly as you would a schema migration.

Integration Note

The canonical key is the cheap first pass of deduplicating addresses by fuzzy canonical key: a GROUP BY _key collapses the bulk of duplicates before any pairwise comparison, leaving only residual near-misses for fuzzy address matching with rapidfuzz. Because step 2 hinges on correct Unicode folding, the NFKC-versus-NFC decision made in Unicode and character normalization in Python directly determines whether visually identical international addresses hash to the same key.