data encoding ai_generated true

File parsing fails or produces unexpected characters at the beginning despite looking correct in editor

ID: data/utf8-bom-invisible-parse-failure

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

UTF-8 BOM (Byte Order Mark, 0xEF 0xBB 0xBF) is prepended by Windows editors (Notepad, Excel export). Most editors hide it, but parsers choke: JSON parsers reject it, CSV first column gets corrupted, shebang lines break.

generic

Workarounds

  1. 95% success Open with encoding='utf-8-sig' in Python (auto-strips BOM if present) 
    open('file.csv', encoding='utf-8-sig')  # strips BOM if present, works normally if absent
  2. 85% success Use dos2unix or sed to strip BOM from files in bulk 
    sed -i '1s/^\xEF\xBB\xBF//' file.csv  # or: dos2unix --remove-bom file.csv
  3. 82% success Configure editors to save as 'UTF-8 without BOM' by default 
    VS Code: 'files.encoding': 'utf8'  # not 'utf8bom'

Dead Ends

Common approaches that don't work:

  1. Open file as UTF-8 and assume clean content 88% fail

    UTF-8 with BOM inserts 3 invisible bytes at the start. json.loads fails. CSV first header gets \ufeff prefix. Dict lookups on first column fail.

  2. Manually strip first 3 bytes from file 72% fail

    Only works if BOM is present. Stripping 3 bytes from non-BOM file corrupts the first character.