How do you explain Unicode normalization and NFC, NFD, NFKC, and NFKD?
Prompt and context
Two strings look identical but compare differently in a database uniqueness check, search, or filename comparison. Explain combining and precomposed characters, the four forms, input and storage boundaries, and why normalization does not replace case folding, language rules, or security policy.
What the interviewer is testing
- Whether you understand canonical versus compatibility equivalence.
- Whether you choose NFC, NFD, NFKC, or NFKD from business semantics.
- Whether you account for Unicode versions, database collation, indexes, and cross-service consistency.
- Whether you can identify information-loss risks from compatibility folding.
Clarifying questions before answering
Confirm whether the field is display text, a login identifier, a search key, a filename, or a security-sensitive identifier; ask about language, case, Unicode version, collation, and whether original input must be retained. A system can keep the original and store a normalized comparison key.
30-second answer framework
Unicode lets one visible text have multiple code-point sequences. NFC performs canonical decomposition followed by composition; NFD performs canonical decomposition. NFKC and NFKD also process compatibility equivalence and may fold characters with distinct formatting or semantic expectations. NFC is a common choice for general text; compatibility folding requires an explicit tolerance for information loss. Fix the Unicode version and keep normalization separate from case folding, script policy, and security checks.
Step-by-step deep dive
1. Canonical equivalence and composition
A precomposed character and a base character plus combining mark can look the same and be canonically equivalent. NFD decomposes them and NFC composes them according to the standard. A normalization form is idempotent: applying it again does not keep changing the string.
2. The cost of compatibility equivalence
NFKD performs compatibility decomposition and NFKC composes afterward. Some font variants, full-width forms, or decorative characters may map to a base representation. Use this for an explicitly broad search or comparison policy, not automatically for passwords, legal text, or content whose appearance must be preserved.
3. Storage and security boundaries
Keep the original input and generate a versioned comparison key. Define uniqueness, tokenization, case folding, and script-confusable checks separately. For login names, domains, or permission identifiers, follow the relevant protocol and security profile; NFKC alone is not a complete defense.
High-quality sample answer
I separate display and comparison values. Unicode permits precomposed characters and combining sequences to represent the same canonical text, so code-point comparison can create a false mismatch. NFD decomposes and NFC recomposes; NFKD and NFKC also apply compatibility mappings, which can discard formatting information. I would normally use NFC for general text and pin the supported Unicode version. A search key may use NFKC only when that loss is intentional, together with case folding and language rules. Store the original, plus a versioned normalized key for database uniqueness. For passwords, signatures, audit text, and security identifiers, I would not add compatibility folding outside the protocol; I would apply the specified identifier and confusable checks. Tests cover combining order, repeated normalization, multilingual input, version upgrades, and collation differences.
Common mistakes
- Saying NFC and NFKC are equivalent.
- Treating normalization as case folding, transliteration, or accent removal.
- Applying NFKC to every field and losing compatibility-character information.
- Normalizing only in the application while ignoring database indexes and service versions.
- Treating visual similarity as security equivalence and missing script confusables.
Follow-up questions and responses
Why not store only the normalized string?
Display, audit, or legal contexts may require the original input. Keeping both avoids irreversible conversion affecting users while supporting stable uniqueness checks.
Can a Unicode version upgrade break uniqueness?
It can affect unassigned code points or normalization data. Record the normalization version, recompute keys offline, check collisions, and migrate indexes in phases.
Does normalization stop homoglyph attacks?
Not completely. Normalization covers defined equivalence relations; security also needs script restrictions, confusable detection, the protocol's identifier profile, and review.