The Same Korean Text, Two Different Byte Sequences

Copy a Korean filename from a Mac, paste it into a search box on a Windows machine, and it finds nothing. The text on screen is identical. The bytes are not.

This isn't a Korean-specific bug, and it isn't an encoding problem in the usual sense — both sides are correctly using UTF-8. It's that Unicode deliberately provides more than one way to write the same character, and the ways are not interchangeable byte-for-byte.

Two ways to spell the same thing

Unicode's model separates a written character from the code points that compose it. Many characters can be written either as a single precomposed code point or as a base character plus combining marks.

The Latin example is é. It exists as U+00E9 (LATIN SMALL LETTER E WITH ACUTE), one code point, and also as U+0065 U+0301 (a plain e followed by COMBINING ACUTE ACCENT), two code points. Both render as é. Both are valid. Compare them as strings and they differ.

Korean has the same duality, structurally deeper. The Hangul Syllables block, U+AC00 through U+D7A3, holds all 11,172 modern syllable blocks as single precomposed code points — 한 is U+D55C, one code point, three bytes in UTF-8. But Unicode also has Hangul Jamo, the individual letters: ᄒ (U+1112), ᅡ (U+1161), ᆫ (U+11AB). Written in sequence they compose into 한, rendering identically, using three code points and nine bytes.

So 한글 is either two code points or six, depending on where the text came from. Both display the same. Both are correct Unicode. Neither == nor a database index nor a search box will consider them equal without help.

The four normalization forms

Unicode's answer is normalization: a defined procedure that rewrites text into one canonical spelling so that equivalent strings become identical strings. There are four forms, generated by two independent choices.

The first choice is composed or decomposed:

• NFD (Normalization Form Decomposition) breaks everything apart into base characters and combining marks. 한 becomes three jamo.
• NFC (Normalization Form Composition) decomposes first, then recomposes everything that has a precomposed form. 한 becomes the single code point U+D55C.

The second choice is canonical or compatibility:

• Canonical equivalence covers characters that genuinely are the same character spelled differently. Precomposed é and decomposed é are canonically equivalent. This transformation is lossless and round-trippable.
• Compatibility equivalence, giving NFKD and NFKC, additionally collapses characters that mean the same thing but carry different formatting. The ligature fi becomes fi. The superscript ² becomes 2. Fullwidth Ａ becomes A. Halfwidth Korean ﾊﾝ becomes normal-width. This transformation loses information and is not reversible.

NFC is the right default. The W3C recommends it for the web, it's what most systems produce, and it's usually the most compact. Use NFKC only when you deliberately want ² and 2 treated as the same — in a search index, or when sanitizing a username so that fullwidth characters can't be used to impersonate someone. Never store NFKC as your canonical copy of user-entered text; you've thrown away distinctions the user may have meant.

Why macOS is the usual culprit

The reason this surfaces so often on Korean and Japanese Macs is a filesystem decision. HFS+ stored filenames in a variant of NFD, decomposing them on the way to disk. APFS relaxed the requirement — it's normalization-insensitive rather than normalization-enforcing — but a great deal of software, and a great many existing files and archives, still carry the decomposed spelling.

Windows and Linux generally store whatever bytes they were handed, and most Windows software produces NFC. So a file named 문서.txt created on a Mac and zipped up can arrive on a Windows machine as 문서.txt — the same characters, decomposed, and often rendered badly because the receiving font stack handles conjoining jamo less gracefully than precomposed syllables.

The symptoms are consistent: file lookups fail even though the name looks right, a git status shows a file as both deleted and added after it moves between machines, a search matches nothing, and a deduplicating script decides two identical filenames are different.

Length, sorting, and everything downstream

Normalization form changes measurable properties of a string, which quietly breaks things that depend on them.

Length. 한글 is 2 code points in NFC and 6 in NFD. In UTF-8 that's 6 bytes versus 18. A VARCHAR(20) that comfortably held a name in NFC can reject the same name in NFD. Any length limit measured in code points or bytes silently means different things for the two forms — a variation on the counting problem described in why counting characters is harder than it looks, and a good reason to normalize before you count characters at all.

Comparison and hashing. Equality, hash maps, database unique constraints, and deduplication all operate on bytes. Two canonically equivalent strings hash differently, so a UNIQUE index will happily accept both spellings of the same name.

Diffing. A text file saved in one form and edited in another shows every touched line as changed even when nothing visible differs. If a text diff reports differences you cannot see, normalization is the first thing to check.

Sorting. Decomposed strings sort by their leading base characters, so NFD text can order differently from NFC text under a naive byte comparison.

Regular expressions. A pattern written against precomposed characters won't match decomposed input. /한/ does not match the three-jamo spelling.

The rule that avoids all of it

Normalize at the boundary. Convert incoming text to NFC the moment it enters your system — form submissions, file uploads, API payloads, filenames read from disk — and store only normalized text. Then every comparison, index, and length check downstream operates on a single consistent spelling.

Every major platform provides this: String.prototype.normalize("NFC") in JavaScript, unicodedata.normalize("NFC", s) in Python, Normalizer.normalize(s, Form.NFC) in Java, text.nfc via ICU in most others.

The failure mode is subtle precisely because nothing looks wrong. There is no mojibake, no replacement characters, no error message — just a search that returns nothing and a filename that doesn't exist, while the screen shows exactly the text you typed.