Generating Canonical Address Keys in Python

TL;DR: A canonical address key is a deterministic hash of a normalized string — lowercase, NFKC-fold, strip punctuation, expand USPS abbreviations, drop unit noise, sort tokens, then blake2b. Two spellings of the same location produce the same key. This page supports deduplicating addresses by fuzzy canonical key, where that key does the cheap bulk of duplicate collapsing before any fuzzy scoring runs.

What Makes a Key “Canonical”

A canonical key must satisfy two properties: determinism (the same input always yields the same key, on every machine and every run) and equivalence collapse (all spellings that denote the same location yield the same key). The first rules out anything order-dependent — Python set iteration, dict ordering across versions, or locale-sensitive casing. The second is what the transform pipeline delivers: each step erases one axis of spurious variance while preserving the information that actually distinguishes addresses.

The hash at the end is not for security; it is for producing a fixed-width, index-friendly, collision-resistant token that is cheap to GROUP BY and to store as a foreign key. blake2b is chosen over md5/sha1 because it is fast, has no deprecation baggage, and lets you set the digest size directly.

Transform Table

Each step targets one specific source of variance. Order is load-bearing: folding Unicode before stripping punctuation prevents accented letters from being discarded, and expanding abbreviations before sorting keeps st and street from sorting to different slots.

# Transform Removes variance in “123 N. Main St., Apt 4” becomes
1 lower() Letter case 123 n. main st., apt 4
2 NFKC normalize Unicode composition, full-width forms (code points folded)
3 Strip combining marks Accents / diacritics 123 n. main st., apt 4
4 Drop unit noise Suite / apartment / floor 123 n. main st.,
5 Strip punctuation ., ,, # 123 n main st
6 Collapse whitespace Repeated spaces 123 n main st
7 Expand abbreviations Suffix / directional spelling 123 north main street
8 Sort tokens Token order (unit before/after street) 123 main north street
9 blake2b hash — (produces the key) a1f3…

The Key Builder

The implementation compiles every regex once at module level, carries explicit type hints, and validates input before doing any work. The USPS map here is abbreviated; a production version loads the full Publication 28 suffix and directional set. See the official USPS Publication 28 for the authoritative abbreviation list.

from __future__ import annotations

import hashlib
import re
import unicodedata
from typing import Dict, Final

# Compile all patterns once at import time — never inside the hot function.
_PUNCT: Final = re.compile(r"[^\w\s]")
_WS: Final = re.compile(r"\s+")
_UNIT: Final = re.compile(
    r"\b(?:apt|apartment|ste|suite|unit|fl|floor|rm|room|bldg|building)\b.*$"
)

# USPS Publication 28 suffixes + directionals (trimmed for illustration).
_ABBREV: Final[Dict[str, str]] = {
    "st": "street", "str": "street", "ave": "avenue", "av": "avenue",
    "blvd": "boulevard", "rd": "road", "dr": "drive", "ln": "lane",
    "ct": "court", "cir": "circle", "pl": "place", "ter": "terrace",
    "hwy": "highway", "pkwy": "parkway", "sq": "square", "trl": "trail",
    "n": "north", "s": "south", "e": "east", "w": "west",
    "ne": "northeast", "nw": "northwest", "se": "southeast", "sw": "southwest",
}


def canonical_string(raw: str, drop_unit: bool = True) -> str:
    """Return a deterministic canonical form of a single-line address.

    Args:
        raw: The raw address string.
        drop_unit: If True, remove secondary-unit designators (per-building grain).

    Returns:
        A normalized, abbreviation-expanded, token-sorted string.

    Raises:
        TypeError: If raw is not a string.
    """
    if not isinstance(raw, str):
        raise TypeError(f"expected str, got {type(raw).__name__}")
    if not raw.strip():
        return ""

    # 1-3: case, Unicode compatibility fold, strip accents.
    text = unicodedata.normalize("NFKC", raw).lower()
    text = "".join(
        c for c in unicodedata.normalize("NFKD", text)
        if not unicodedata.combining(c)
    )
    # 4: drop secondary-unit noise before punctuation removal.
    if drop_unit:
        text = _UNIT.sub("", text)
    # 5-6: strip punctuation, collapse whitespace.
    text = _WS.sub(" ", _PUNCT.sub(" ", text)).strip()
    # 7: expand abbreviations token by token.
    tokens = [_ABBREV.get(tok, tok) for tok in text.split()]
    # 8: deterministic token order.
    tokens.sort()
    return " ".join(tokens)


def canonical_key(raw: str, drop_unit: bool = True) -> str:
    """Hash the canonical string into a stable 16-byte hex key.

    Empty or whitespace-only input yields a fixed sentinel key so that
    blank records cluster together instead of raising.
    """
    canon = canonical_string(raw, drop_unit=drop_unit)
    return hashlib.blake2b(canon.encode("utf-8"), digest_size=16).hexdigest()

Note that the unit regex is applied before punctuation stripping so that Apt 4, is removed cleanly with its trailing comma. Reordering these two steps would leave a dangling comma token that pollutes the key.

Vectorized pandas Example

For batch tables, avoid a per-row Python call where possible, but since the transform is string-heavy the practical pattern is a single .map over the column, then a vectorized hash. Compute the canonical string once and reuse it — it feeds both the key and any downstream rapidfuzz scoring.

import hashlib

import pandas as pd


def add_canonical_key(df: pd.DataFrame, addr_col: str) -> pd.DataFrame:
    """Attach canonical string and canonical key columns to a DataFrame.

    Args:
        df: Input frame containing an address column.
        addr_col: Name of the raw address column.

    Returns:
        A copy with '_canon' and '_key' columns added.
    """
    out = df.copy()
    out["_canon"] = out[addr_col].fillna("").astype(str).map(canonical_string)
    out["_key"] = out["_canon"].map(
        lambda c: hashlib.blake2b(c.encode("utf-8"), digest_size=16).hexdigest()
    )
    return out


# Deduplicate: one survivor per canonical key.
def collapse(df: pd.DataFrame, addr_col: str) -> pd.DataFrame:
    """Collapse exact canonical-key duplicates, keeping the first row per key."""
    keyed = add_canonical_key(df, addr_col)
    return keyed.drop_duplicates(subset="_key", keep="first")

Edge Cases

Empty and Non-String Input

Real tables carry nulls, integers coerced from spreadsheets, and whitespace-only cells. canonical_string raises TypeError on non-strings so silent coercion never corrupts a key, and returns "" for blank input so blank rows share a single sentinel key rather than each hashing to a different value. In pandas, fillna("").astype(str) normalizes the column before mapping.

Grain Determines Unit Handling

With drop_unit=True, 123 Main St Apt 4 and 123 Main St Apt 9 produce the same key (per-building grain). With drop_unit=False, the unit token survives, is abbreviation-normalized, and discriminates the two. Choose per pipeline and keep it consistent; mixing grains within one key column produces silent over- or under-merging. The trade-off is discussed in deduplicating addresses by fuzzy canonical key.

Versioning the Key Function

Any change to the abbreviation map or transform order changes the output key for some inputs, invalidating every previously stored key. Treat the function as versioned: embed a version tag alongside the stored key (for example v2:a1f3…) and recompute in a controlled backfill when the logic changes, exactly as you would a schema migration.

Integration Note

The canonical key is the cheap first pass of deduplicating addresses by fuzzy canonical key: a GROUP BY _key collapses the bulk of duplicates before any pairwise comparison, leaving only residual near-misses for fuzzy address matching with rapidfuzz. Because step 2 hinges on correct Unicode folding, the NFKC-versus-NFC decision made in Unicode and character normalization in Python directly determines whether visually identical international addresses hash to the same key.