Removing Diacritics From Addresses in Python

TL;DR: Decompose with NFKD, drop the combining marks, then apply a small table of language-specific substitutions that decomposition cannot handle — ß, ø, ł, đ. Store the result as a match key beside the original, never in place of it. The surrounding rules are in Unicode and character normalization in Python.

The Function

from __future__ import annotations

import unicodedata

# Characters that NFKD cannot decompose because they are letters in their own
# right, not a base letter plus a mark. Each needs an explicit mapping.
EXPLICIT: dict[str, str] = {
    "ß": "ss", "ẞ": "SS",
    "ø": "o", "Ø": "O",
    "æ": "ae", "Æ": "AE",
    "œ": "oe", "Œ": "OE",
    "ł": "l", "Ł": "L",
    "đ": "d", "Đ": "D",
    "ð": "d", "Ð": "D",
    "þ": "th", "Þ": "TH",
    "ı": "i", "İ": "I",
}

_TRANSLATION = str.maketrans(EXPLICIT)


def fold_to_ascii(text: str) -> str:
    """Return an ASCII match key. NOT a delivery address."""
    text = text.translate(_TRANSLATION)
    decomposed = unicodedata.normalize("NFKD", text)
    stripped = "".join(ch for ch in decomposed if not unicodedata.combining(ch))
    return stripped.encode("ascii", "ignore").decode("ascii")

Order matters: the explicit substitutions run before decomposition, because some of them produce multi-character results that themselves want normalising, and running them afterwards would leave the decomposed marks in place around the substituted letters.

Why Decomposition Alone Is Not Enough

The failure mode in the closing note is worse than it first appears. Straße folding to Strae does not merely fail to match Strasse; it produces a key that matches nothing at all, so the record silently becomes its own singleton in every deduplication run.

The Fold Is a Key, Not an Address

Every rule here applies to the matching key. The delivery address keeps its diacritics, because they are part of the correct spelling of the street and because some postal operators — and every recipient — care about the difference.

The last row is the one that generates real arguments. A partner whose system is ASCII-only forces a lossy export, and the right approach is to send the folded form for that partner while keeping the original internally — not to fold at ingestion and lose it for everyone.

German and Nordic Cases Deserve Care

ß is the most consequential single character in European address data, because German postal convention treats ß and ss as equivalent while a naive fold produces neither. Python’s str.lower() leaves ß intact and str.casefold() converts it to ss, which is why casefold rather than lower is the correct operation when building a key.

Scandinavian ø and å raise a different question: ø folds naturally to o, but å is traditionally written aa in Danish, and a fold to a will not match an older record spelled the traditional way. Where a corpus spans decades, adding å → aa as an alternate key is more effective than choosing one and hoping.

Polish ł is a stroked l rather than an accented one, which is exactly why decomposition misses it, and Turkish dotless ı interacts badly with case folding — I.lower() is i in most locales and ı in Turkish. Since address matching should never be locale-sensitive, map both to i explicitly and take the small loss.

Measuring the Fold

Running this comparison on your own corpus is a half-hour exercise and settles the question of whether folding is worth the extra column. On predominantly Anglophone data the answer is usually that it costs little and gains little; on European data it is frequently the single largest improvement available to a deduplication pipeline.

Testing the Fold

A folding function is unusually easy to test well, because its behaviour is fully specified by a table of inputs and outputs and because the interesting cases are all short strings.

Build the suite from four groups. The first is one row per entry in the explicit table, asserting the exact substitution — this catches the case where someone edits the dictionary and reverses a mapping. The second is a set of accented Latin letters covering the marks that appear in your markets, confirming that decomposition and mark removal are working together. The third is a set of strings that must pass through unchanged, so that a regression which starts mangling plain ASCII is caught immediately.

The fourth group is the interesting one: real address strings from each market, with their expected folded form written out. These are the tests that fail when someone reorders the substitution and decomposition steps, because the failure only shows up on a string that needs both — Straße is the canonical example, and a handful like it are worth keeping.

Add a property test if the toolchain supports one: for any input, the output must contain only ASCII characters, and folding an already-folded string must be a no-op. Both properties are cheap to check across thousands of generated inputs and both catch classes of bug that example-based tests miss, particularly around unusual scripts and combining sequences that nobody thought to enumerate.

Finally, keep a golden file of the folded forms for a sample of production addresses and diff it on every change. That file is the fastest way to see the blast radius of an edit to the table, and it makes a deliberate change reviewable rather than a leap of faith.

Edge Cases and Failure Modes

Folding non-Latin scripts. ASCII folding of Greek, Cyrillic or CJK text deletes it entirely. Detect the script before folding and skip it, or key those records on the NFKC form instead — losing the whole address is far worse than keeping a non-ASCII key.

Double folding. Applying the fold to an already-folded string is harmless but wasteful, and it is a sign the pipeline has lost track of which column holds which form. Name the columns distinctly — street and street_key — so the mistake is visible in code review.

Folding before parsing. Diacritics occasionally carry parsing signal, particularly in Spanish ordinals (2ºD) and Portuguese. Parse first, fold second, so the parser sees the original.

Assuming the fold is reversible. It is not: ss could have been ß or ss. Never reconstruct a display value from the key, which is the practical reason the original must be stored rather than derived.

Keep the golden file small enough that a human will actually read the diff — a few hundred representative rows beats a few thousand exhaustive ones, because a diff nobody reads provides exactly as much protection as no diff at all.

One further check worth automating: assert that the folded key for every row in a production sample is non-empty. A key that folds to the empty string means the input was entirely non-Latin and the fold destroyed it, which is a routing bug rather than a folding bug — and it is far easier to catch with a one-line assertion than by noticing that a set of records never deduplicates against anything.

Two Columns, Two Contracts

The rule that the fold is a key and not a value is easiest to hold if the schema states it, so the two columns are named and typed for their purposes rather than left to convention.

Integration Note

The folded key is what canonical address keys are built from, and it must therefore be covered by the same version discipline: a change to the EXPLICIT table changes every key derived from it. Bumping the normalisation version alongside any edit to that dictionary is what keeps the cache and the deduplication tables consistent, as described in versioning cache keys across normalisation changes.

Related

• Unicode and Character Normalization in Python — the four normalisation forms and when each applies.
• Handling Special Characters in Global Address Data — the allow-list and mojibake repair that run before folding.
• Generating Canonical Address Keys in Python — where the folded form is consumed.