Why Your JSON Fails to Parse: Zero-Width Spaces Copied from ChatGPT
You ask ChatGPT for a config snippet. You copy it, paste it into your editor, and compare it with the file that works. The two look identical, character for character. Then you run it. In Node, a pasted file that starts with a byte order mark gives you this (the invisible character is written here
You ask ChatGPT for a config snippet. You copy it, paste it into your editor, and compare it with the file that works. The two look identical, character for character. Then you run it.
In Node, a pasted file that starts with a byte order mark gives you this (the invisible character is written here as <U+FEFF>; your terminal prints nothing at that position):
SyntaxError: Unexpected token '<U+FEFF>', "<U+FEFF>{"name": "x"}" is not valid JSON Enter fullscreen mode Exit fullscreen mode In Python, a zero-width space pasted into source code gives you this:
SyntaxError: invalid non-printable character U+200B Enter fullscreen mode Exit fullscreen mode The BOM variant of the Python error is the same message with U+FEFF at the end. In both languages the message names a character you cannot see, and the line it points to looks fine. The string you pasted is not the string you think it is.
These are the code points that cause most of the trouble:
Why do AI chat interfaces leave them behind? The usual explanation is that the chat UI builds its output incrementally and renders it as rich text, and some of that structure ends up in the clipboard when you copy. These are rendering and streaming artifacts, not watermarks. Nothing about them is a hidden signal, and they are not tied to the content of the answer. The same characters show up when you copy from PDFs, Slack, Notion and ordinary web pages.
One more thing worth knowing: a zero-width space inside a JSON string is valid JSON. JSON.parse accepts it. The failure is later, when config["name"] returns undefined because the key is really "name\u200B". A parse error only happens when the character lands outside a string, or when it is a BOM at the start.
cat -A file.json # BOM shows as M-oM-;M-?, U+200B as M-bM-^@M-^K xxd file.json | head # a BOM is the bytes ef bb bf at offset 0 grep -nP '[\x{200B}\x{FEFF}\x{00A0}]' file # needs a UTF-8 locale Enter fullscreen mode Exit fullscreen mode If grep -P complains that a code point is too large, run it with LC_ALL=C.UTF-8.
s = open ( " file.json ", encoding = " utf-8 " ). read () print ( repr ( s )) # repr escapes U+200B as \u200b print ([ hex ( ord ( c )) for c in s if ord ( c ) > 0x7e ]) Enter fullscreen mode Exit fullscreen mode Node:
const s = require ( " fs " ). readFileSync ( " file.json ", " utf8 " ); console. log ([... s ]. map ( c => c. codePointAt ( 0 ). toString ( 16 )). filter ( h => parseInt ( h, 16 ) > 0x7e )); Enter fullscreen mode Exit fullscreen mode Editor and git:
Strip the characters, but handle two cases separately. NBSP should become a normal space, not disappear, or a\u00A0b turns into ab. And U+200D must survive when it joins emoji, because that is how a family emoji is built: person, ZWJ, person, ZWJ, child. Remove every ZWJ and the family falls apart into three separate emoji.
const KEEP = " \ uE000 "; // private-use placeholder, not present in normal text function cleanInvisible ( text ) { return text // protect a ZWJ that sits between two emoji. replace ( / (\p {Extended_Pictographic} \u FE0F?)\u 200D (?=\p {Extended_Pictographic} ) /gu, `$1 ${ KEEP } ` ) // remove zero-width chars, direction marks, word joiner, BOM, soft hyphen. replace ( / [\u 200B- \u 200F \u 2060 \u FEFF \u 00AD ] /g, "" ) // NBSP becomes a regular space. replace ( / \u 00A0/g, " " ) // restore the protected joiners. replaceAll ( KEEP, " \ u200D " ); } const family = " \ u{1F468} \ u200D \ u{1F469} \ u200D \ u{1F467} "; console. log ( cleanInvisible ( family ) === family ); // true console. log ( cleanInvisible ( " a \ u200Bb \ u00A0c \ uFEFF " )); // "ab c" Enter fullscreen mode Exit fullscreen mode Python's re has no \p{...}, so this version approximates "emoji" with two common ranges. Adjust them to your data:
import re EMOJI = r " [\U0001F300-\U0001FAFF\u2600-\u27BF] " KEEP = " \uE000 " def clean_invisible ( s: str ) -> str: s = re. sub ( rf " ( { EMOJI } \uFE0F?)\u200D(?= { EMOJI } ) ", lambda m: m. group ( 1 ) + KEEP, s ) s = re. sub ( " [ \u200B - \u200F\u2060\uFEFF\u00AD ] ", "", s ) s = s. replace ( " \u00A0 ", " " ) return s. replace ( KEEP, " \u200D " ) Enter fullscreen mode Exit fullscreen mode Shell, using perl for the emoji-aware version and sed for the blunt one: