CSV_ENCODING_WINDOWS_1252 data encoding_error ai_generated true

CSV file encoded with Windows-1252 causes garbled text when read as UTF-8

ID: data/csv-encoding-mismatch-windows-1252

Also available as: JSON · Markdown · 中文
95%Fix Rate
90%Confidence
1Evidence
2023-05-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Python 3.10 active — — —
pandas 2.0.0 active — — —
Microsoft Excel 2019 active — — —

Root Cause

The CSV file was saved with Windows-1252 encoding (common on legacy systems) but the reader assumes UTF-8, causing misinterpretation of bytes for characters like accented letters.

generic

中文

CSV文件使用Windows-1252编码保存（在旧系统中常见），但读取器假定为UTF-8，导致对重音字母等字符的字节错误解释。

Official Documentation

https://docs.python.org/3/library/codecs.html#standard-encodings

Workarounds

  1. 95% success Use Python's `chardet` library to detect encoding and then read with the correct encoding: `import chardet; with open('file.csv', 'rb') as f: result = chardet.detect(f.read()); df = pd.read_csv('file.csv', encoding=result['encoding'])`. 
    Use Python's `chardet` library to detect encoding and then read with the correct encoding: `import chardet; with open('file.csv', 'rb') as f: result = chardet.detect(f.read()); df = pd.read_csv('file.csv', encoding=result['encoding'])`.
  2. 90% success Convert the file to UTF-8 using iconv: `iconv -f WINDOWS-1252 -t UTF-8 input.csv > output.csv`. 
    Convert the file to UTF-8 using iconv: `iconv -f WINDOWS-1252 -t UTF-8 input.csv > output.csv`.

中文步骤

  1. Use Python's `chardet` library to detect encoding and then read with the correct encoding: `import chardet; with open('file.csv', 'rb') as f: result = chardet.detect(f.read()); df = pd.read_csv('file.csv', encoding=result['encoding'])`.
  2. Convert the file to UTF-8 using iconv: `iconv -f WINDOWS-1252 -t UTF-8 input.csv > output.csv`.

Dead Ends

Common approaches that don't work:

  1. 90% fail

    Adding UTF-8 BOM to the file does not fix the encoding; it only helps with byte order marking but does not convert the actual character encoding.

  2. 80% fail

    Manually replacing garbled characters one by one is error-prone and does not scale for large files.