python encoding_error ai_generated true

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

ID: python/pandas-csv-unicode-decode

Also available as: JSON · Markdown · 中文
80%Fix Rate
90%Confidence
0Evidence
2024-02-19First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
3.x active — — —

Root Cause

pd.read_csv defaults to UTF-8, but the file is encoded in UTF-16 (BOM 0xFF 0xFE), Latin-1, or CP1252. 0xff at position 0 is the UTF-16 LE BOM.

generic

中文

pd.read_csv 默认使用 UTF-8，但文件编码为 UTF-16（BOM 0xFF 0xFE）、Latin-1 或 CP1252。位置 0 的 0xff 是 UTF-16 LE BOM。

Workarounds

  1. 95% success 
    pd.read_csv(path, encoding='utf-16')  # or 'latin-1' / 'cp1252'
  2. 90% success 
    Detect with `import chardet; enc = chardet.detect(open(path,'rb').read(10000))['encoding']`

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Silently corrupts non-ASCII text

  2. 90% fail

    Same decode error on open()