For data analysts
CSV & JSON corrupted by invisible characters
An export that "looks fine" refuses to import. Two IDs that should match don't. A column that shouldn't exist keeps appearing. Often the cause is invisible characters — and they're easy to miss, hard to blame, and quick to fix once you know.
By NoAtMark · Published Aug 12, 2026 · 8 min read
The data problem in practice
You pull a customer export from a tool, load it into a data warehouse, and a scheduled job starts dropping rows. You run a LEFT JOIN on a key — and a quarter of the rows silently don't match. You check the file in a spreadsheet: everything looks identical. It takes hours — or a hexdump — to find that the values differ by an invisible U+200B.
Invisible characters enter data files the same way they enter everything: copying from web pages, AI tools, chat, or files that were themselves assembled from pasted text. Each hop can inject zero-width characters, BOMs, soft hyphens, or special spaces into cell values and JSON keys.
Why CSV and JSON are especially vulnerable
CSV and JSON are plain-text formats: every byte matters. A hidden character in a value is still a character:
- A zero-width space inside a numeric-looking ID makes it a string and breaks numeric joins.
- A BOM at the start of a file can make a JSON parser reject the whole document (the BOM must be stripped before parsing).
- A hidden character in a header becomes a column with a name you can't see.
Data systems validate strictly — a value with an invisible byte is simply not equal to a value without it.
The symptoms analysts hit
| Symptom | Likely cause |
|---|---|
| Import fails or rows are rejected | Invisible byte in a value or a BOM at file start |
| Join key doesn't match | Key differs by invisible characters on one side |
DISTINCT shows "duplicates" | Two values differ only by invisible bytes |
| A mystery column appears | Invisible byte in a header name |
| JSON parse error on a "valid" file | BOM or invisible chars before/inside tokens |
How to fix it — without breaking the structure
The critical rule for CSV/JSON is: remove invisible characters only — keep every comma, quote, bracket, and line break. That's exactly what the file cleaner does for CSV and JSON: it strips invisible code points and leaves your delimiters and record boundaries untouched.
- Drop your
.csvor.jsoninto the file cleaner. - Read the scan report — it shows every invisible character found, grouped by type.
- Clean and download the fixed file. Columns, rows, and structure are preserved.
For teams, the text hygiene API can add the same cleaning step to an ETL pipeline so exports are clean before they reach the warehouse.
Prevention in your pipeline
- Clean at the source. Add a cleaning step to exports that pass through web forms, AI tools, or manual copy-paste.
- Normalise keys. Where practical, strip invisible characters from IDs and join keys on write.
- Investigate "phantom" mismatches before the database. When data looks equal but isn't, check for invisible bytes first — it's faster than querying around the bug.
- Validate headers. A quick check for invisible bytes in column names prevents mystery columns.
Honest caveats
- Cleaning invisible characters won't fix malformed CSV quoting or genuinely wrong delimiters — those are different problems.
- Always review the scan report: in rare cases, a legitimate zero-width character (for example, in an emoji sequence) could be intentional. In data files it almost never is.
Frequently asked questions
Can a spreadsheet show invisible characters?
Sometimes — but unreliable. A hexdump is definitive; a scan tool is more practical. The file cleaner shows counts grouped by character type.
Will cleaning break my quoting or multiline fields?
No. The cleaner only removes invisible code points; it doesn't touch delimiters, quotes, or line breaks. Your structure stays byte-for-byte identical apart from the removed characters.
Is this caused by AI tools specifically?
Not exclusively — but AI output and copy-paste from the web are common sources. Cleaning exports that passed through those channels is a cheap insurance.
References & further reading
- Unicode Standard (code points & semantics) — unicode.org
- CSV is a text format — every byte matters — RFC 4180
- Our developer guide to invisible characters — noatmark.com/blog/invisible-characters-in-code
NoAtMark is an independent project focused on text hygiene for the AI age. This article is written and fact-checked by the team; sources are linked under References. Every tool mentioned is free, private, and runs in your browser. About us →
Stop chasing phantom mismatches
Add a 30-second clean to your data imports. Free, private, in-browser.