Handling Special Characters in Global Address Data

TL;DR: Apply unicodedata.normalize('NFC', text), strip invisible control characters, expand Latin ligatures, and collapse whitespace before submitting addresses to any geocoding API. This four-step sequence — part of the Unicode and Character Normalization in Python workflow — eliminates the silent mismatches that cause duplicate geocoded records and failed coordinate lookups.

---

Why Special Characters Break Geocoding Pipelines

Global address datasets rarely arrive clean. A single street name like café du Marché may appear in your pipeline as six structurally different byte sequences depending on its source system: CRM export, municipal open data, e-commerce checkout, or legacy CSV dump. Geocoding APIs and spatial databases expect strict UTF-8 in NFC form. When a pipeline passes NFD strings, BOM markers, or zero-width spaces, parsers either reject the payload, return null coordinates, or silently create duplicate records for the same physical location.

Four failure modes account for the majority of production incidents:

The diagram below shows where normalization must sit in the pipeline — before field extraction, not after.

---

A Character Budget for Address Fields

Rather than enumerating characters to remove, it is more robust to define what is allowed through and treat everything else as an exception to be logged. Address data legitimately contains letters from many scripts, digits, spaces, hyphens, apostrophes, periods, commas and slashes — and essentially nothing else. Control characters, zero-width joiners and emoji are always noise in this domain.

The counters in the right-hand panel earn their keep during incidents. When a partner switches export tooling, the signature is almost always a step change in one of these categories rather than a drop in parse rate — the records still parse, they simply carry an invisible passenger that breaks the join two systems downstream.

The Normalization Spec

The four operations must run in this exact order. Changing the sequence produces different results: NFC after ligature expansion leaves expanded pairs in decomposed state; whitespace collapse before control stripping misses spaces introduced by removed invisible chars.

---

Minimal Runnable Implementation

import unicodedata
import re
from typing import Union
import pandas as pd

# Compile once at module level — never inside a loop or function body.
# Targets: C0/C1 control codes, zero-width chars U+200B–U+200F,
# directional marks U+202A–U+202E, BOM U+FEFF, and soft hyphen U+00AD.
_CONTROL_RE: re.Pattern[str] = re.compile(
    r'[\x00-\x1F\x7F-\x9F­​-‏‪-‮]'
)

# Latin compatibility ligatures U+FB00–U+FB06 (ff, fi, fl, ffi, ffl, st, st)
_LIGATURE_RE: re.Pattern[str] = re.compile(r'[ff-st]')


def normalize_address(raw: Union[str, bytes, None], ascii_fallback: bool = False) -> str:
    """
    Normalize a global address string for geocoding API submission.

    Steps applied in order:
      1. Input coercion (bytes → str, None → '')
      2. NFC canonical composition
      3. Control and invisible character removal
      4. Latin ligature expansion (NFKD on matched ligatures only)
      5. Whitespace collapse
      6. ASCII fallback (optional, lossy — log when triggered)

    Args:
        raw: Raw address value; accepts str, bytes, or None.
        ascii_fallback: When True, strips all non-ASCII characters.
                        Use only for legacy APIs that reject UTF-8.

    Returns:
        A normalized, NFC-encoded str safe for geocoder submission.
    """
    if raw is None:
        return ''
    if isinstance(raw, bytes):
        raw = raw.decode('utf-8', errors='replace')
    elif not isinstance(raw, str):
        raw = str(raw)

    # Step 2: canonical composition
    text: str = unicodedata.normalize('NFC', raw)

    # Step 3: strip control and invisible characters
    text = _CONTROL_RE.sub('', text)

    # Step 4: expand ligatures to base character pairs
    text = _LIGATURE_RE.sub(
        lambda m: unicodedata.normalize('NFKD', m.group(0)),
        text,
    )

    # Step 5: collapse whitespace
    text = re.sub(r'\s+', ' ', text).strip()

    # Step 6 (optional, lossy): ASCII-only fallback
    if ascii_fallback:
        text = unicodedata.normalize('NFKD', text)
        text = text.encode('ascii', 'ignore').decode('ascii')

    return text


# Vectorized variant for pandas — avoid Python-level loops on large frames
def normalize_address_series(
    series: pd.Series,
    ascii_fallback: bool = False,
) -> pd.Series:
    """
    Apply normalize_address to a pandas Series of raw address values.

    Args:
        series: Series of raw address strings (may contain NaN).
        ascii_fallback: Passed through to normalize_address.

    Returns:
        Series of normalized strings with NaN replaced by empty string.
    """
    return series.fillna('').apply(
        lambda v: normalize_address(v, ascii_fallback=ascii_fallback)
    )

---

Pattern Breakdown

Control character regex

[\x00-\x1F\x7F-\x9F­​-‏‪-‮]

Ligature regex

[ff-st]

This range covers the seven Latin compatibility ligatures (ff ff, fi fi, fl fl, ffi ffi, ffl ffl, ſt st, st st). Applying NFKD inside the substitution lambda decomposes each ligature to its constituent ASCII letters without affecting any other character in the string — in particular, without touching valid diacritics like ü or ñ.

---

Mojibake Has a Recognisable Fingerprint

The most common encoding defect in address data is not exotic Unicode; it is UTF-8 bytes that were decoded as Latin-1 somewhere upstream. It has a distinctive signature — a sequence beginning Ã or â€ — and, importantly, it is often reversible if you catch it before anything else touches the string.

Run the repair once, at ingestion, and record how many rows it fired on. That count is a data-quality metric about your suppliers, not your parser, and it belongs on the same dashboard as parse rate — a partner whose mojibake rate jumps from zero to two percent has changed something, and you want to hear about it from a chart rather than from a customer.

Edge Cases and Failure Modes

1. CJK full-width digits in Japanese addresses

Japanese addresses frequently contain full-width ASCII digits (１２３ U+FF11–U+FF19) for house numbers. The normalize_address function above preserves these because NFC does not decompose compatibility forms. If a downstream parser expects ASCII digits, apply unicodedata.normalize('NFKC', text) as an additional step only to the house-number field extracted after splitting — not to the whole address string. Global NFKC collapses superscripts and fractions, which breaks addresses in some European locales.

import unicodedata

def normalize_digits(field: str) -> str:
    """Normalize full-width digits to ASCII in a single address field."""
    return unicodedata.normalize('NFKC', field)

# Apply only after field extraction, not to the raw string:
house_number = normalize_digits(parsed["house_number"])

2. Arabic presentation forms in Gulf address data

Gulf-region datasets from municipal portals often contain Arabic presentation forms (U+FE70–U+FEFF) — isolated, initial, medial, and final glyphs rather than the base Unicode code points. NFC does not normalize these. Use NFKC on Arabic-script fields specifically, or apply the arabic-reshaper + python-bidi libraries when visual shaping is also required.

def normalize_arabic_field(field: str) -> str:
    """Normalize Arabic presentation forms to base code points."""
    return unicodedata.normalize('NFKC', field)

3. Mixed Windows-1252 / UTF-8 streams

Legacy CRM exports often inject Windows-1252 bytes into nominally UTF-8 streams. The bytes.decode('utf-8', errors='replace') call in step 1 converts undecodable bytes to U+FFFD (replacement character <?>) rather than crashing. Log these occurrences: a high replacement-character rate signals that the source encoding declaration is wrong and the entire batch needs re-ingestion with chardet-detected encoding.

import chardet

def safe_decode(raw: bytes) -> str:
    """Detect encoding and decode; fall back to UTF-8 with replacement."""
    detected = chardet.detect(raw)
    encoding = detected.get('encoding') or 'utf-8'
    return raw.decode(encoding, errors='replace')

---

Integration Note

normalize_address belongs at the ingestion boundary of your Unicode and Character Normalization in Python pipeline — immediately after CSV/JSON parsing and before any field extraction, tokenization, or API call. Applying it later (inside the geocoding client, for example) allows malformed strings to corrupt intermediate data structures and makes debugging harder.

For distributed workloads on Airflow, Spark, or Kafka Streams, broadcast the two compiled regex patterns as module-level constants — never call re.compile() inside a task function or a UDF, as it adds unnecessary compilation overhead per row. When building async geocoding requests in Python, normalize the entire batch synchronously before fanning out to the async layer; mixing normalization into async tasks complicates error attribution.

---

### Why does NFC miss soft hyphens and zero-width spaces?

NFC is a canonical equivalence operation: it only touches characters that represent the same abstract character through different code point sequences. Soft hyphens (U+00AD) and zero-width spaces (U+200B) are distinct characters in their own right — they are not alternate representations of anything else. The control-character regex is what removes them.

### Is it safe to run normalize_address on non-Latin scripts?

Yes for NFC, with one caution: do not enable ascii_fallback=True on Cyrillic, Greek, CJK, or Arabic scripts. The fallback drops every non-ASCII character, turning a valid address into an empty string or meaningless fragment. For transliteration of non-Latin scripts, use unidecode or the transliterate library, and treat the transliterated form as a search key only — never overwrite the canonical stored value.

### Should normalization run before or after deduplication hashing?

Before. Two addresses that are canonically identical must produce the same hash for deduplication to work. If you hash before NFC + invisible-char stripping, NFD and NFC versions of the same address produce different hashes and survive as duplicates. Normalize first, then hash.

---

Related

• Unicode and Character Normalization in Python — the parent reference covering all four normalization forms, CJK full-width variants, Arabic presentation forms, and locale-specific casing rules.
• Core Address Parsing & Standardization — the top-level guide to the full address pipeline from raw ingestion through validated, geocodable output.
• Normalizing International Addresses with libpostal — how to parse and standardize global address components after character normalization is complete.
• How to Parse Street Numbers and Suffixes with Regex — regex patterns that consume the normalized string to extract house numbers, directionals, and street type suffixes.