Why Your CSV Opens as Mojibake in Excel

You export a CSV with Korean text, open it in Excel, and every Hangul character has become something like 문ìž. The file is fine — open it in a text editor and the content is intact. Excel simply decoded the bytes using the wrong rulebook, and nothing in the file told it which one to use.

Bytes are not characters

A file on disk holds bytes. Turning bytes into characters requires knowing which encoding was used, and there is no way to determine that from the bytes alone with certainty. It can only be declared, or guessed.

Some formats carry the declaration inside them. HTML has <meta charset="utf-8">. XML has a declaration on the first line. Both of those are a slight bootstrap paradox — you must read some bytes to find the declaration — resolved by requiring the declaration to appear early and in ASCII.

CSV has no such field. RFC 4180 doesn't define one. A CSV file is bytes with commas and newlines in it. The encoding lives entirely in out-of-band knowledge: what the exporting system used, and what the importing system assumes.

Mojibake is what you see when those two disagree. The name is Japanese — 文字化け, roughly "character transformation" — and it describes a decoding mismatch, not damage. The bytes are unchanged and the original text is fully recoverable by decoding again with the right encoding.

The encodings involved

ASCII defines 128 characters in 7 bits. It covers English and nothing else, and every encoding below is designed to be compatible with it in the low range.

EUC-KR and CP949 encode Korean. EUC-KR uses two bytes per Hangul syllable and covers 2,350 of them — the ones deemed common in 1987. CP949, Microsoft's extension (also called UHC), keeps EUC-KR compatibility and adds the remaining syllables to reach all 11,172. Windows in a Korean locale has historically treated "ANSI" as CP949, and a great deal of Korean software still writes it. Japan has Shift-JIS, China has GB18030, Western Europe has CP1252, and each is mutually unintelligible with the others.

UTF-8 encodes all of Unicode using one to four bytes per code point. ASCII characters remain a single byte with their original values, which is why UTF-8 slid into the ASCII world so easily. A Hangul syllable takes three bytes. It's now the encoding of essentially the entire web, and the correct default for anything new.

The mojibake pattern tells you what happened. Korean text showing as 문ìž means UTF-8 bytes were decoded as a single-byte Western encoding — each three-byte syllable became three separate accented Latin characters. Korean text showing as ? or � means the decoder recognized the bytes as invalid and substituted a replacement character, which is lossier: those bytes are gone and re-decoding won't bring them back.

Excel's guess, and the BOM

Excel's CSV import is the usual point of failure. Historically, double-clicking a .csv on a Korean or Japanese Windows machine made Excel decode it with the system ANSI codepage — CP949, not UTF-8. Recent versions are better, but the behavior still varies by version, locale, and whether the file was opened by double-click or through the import wizard.

The workaround the ecosystem settled on is the UTF-8 BOM: three bytes, EF BB BF, at the very start of the file. In UTF-16 a byte order mark genuinely indicates byte order; in UTF-8 there is no byte order to mark, so it functions purely as a signature saying "the following is UTF-8." Excel honors it and decodes correctly.

The BOM is a mixed blessing, which is worth knowing before you add it reflexively. Many Unix tools don't strip it, so those three bytes end up glued to the first column name — a header that looks like id but is actually id, which fails an exact-match lookup in a way that is genuinely hard to see. Some JSON and YAML parsers reject a leading BOM outright. Shell scripts with a BOM before #! don't execute.

So: add the BOM for files a human will open in Excel, omit it for files a program will parse. If you're producing one export for both audiences, the BOM is usually the lesser problem, but the consuming code should strip it defensively.

Doing the export properly

• Write UTF-8. Not CP949, even for a Korean audience. CP949 can't represent characters outside its repertoire, so a name with an unusual character or a comment with an emoji is unrepresentable and will be dropped or mangled at write time — a real loss, not a display issue.
• Add the BOM if Excel is a target. It costs three bytes and removes the most common support question an export feature generates.
• Set the charset on downloads: Content-Type: text/csv; charset=utf-8. Browsers respect it, and it costs nothing.
• Consider offering .xlsx instead. The xlsx format is a ZIP of XML files that declare their own encoding, so there's no guessing at all. If most recipients open the file in Excel anyway, giving them the format Excel actually wants sidesteps the entire problem.

Recovering a file that's already mangled

If the bytes on disk are still the original ones and only the display is wrong, nothing is lost. In Excel, use Data → Get Data → From Text/CSV and choose "65001: Unicode (UTF-8)" in the File Origin dropdown rather than double-clicking the file. On the command line, iconv -f cp949 -t utf-8 in.csv > out.csv converts between encodings directly.

If the file was saved after being decoded wrongly, the damage may be permanent — the mangled characters were written back as real characters, and any byte the decoder replaced with � is unrecoverable. Round-tripping is sometimes possible when the wrong encoding happened to be lossless for those bytes, but it's not something to rely on.

And the other CSV problems

Encoding is only the failure that's most visible. A CSV also has no declared delimiter — locales that use a comma as the decimal separator often use a semicolon instead — no type information, so leading zeros in postal codes and long ID numbers get silently reinterpreted as numbers, and no agreement on quoting or line endings. Those are covered in the CSV edge cases that break naive converters, and they're the reason converting CSV to JSON is more involved than splitting on commas.

CSV's durability comes from having almost no specification, which makes it trivial to write. The bill for that arrives on the reading side, and encoding is the first line item.