CSV_ENCODING_WINDOWS_1252 data encoding_error ai_generated true

使用Windows-1252编码的CSV文件在按UTF-8读取时导致乱码

CSV file encoded with Windows-1252 causes garbled text when read as UTF-8

ID: data/csv-encoding-mismatch-windows-1252

其他格式: JSON · Markdown 中文 · English
95%修复率
90%置信度
1证据数
2023-05-10首次发现

版本兼容性

版本状态引入弃用备注
Python 3.10 active — — —
pandas 2.0.0 active — — —
Microsoft Excel 2019 active — — —

根因分析

CSV文件使用Windows-1252编码保存（在旧系统中常见），但读取器假定为UTF-8，导致对重音字母等字符的字节错误解释。

English

The CSV file was saved with Windows-1252 encoding (common on legacy systems) but the reader assumes UTF-8, causing misinterpretation of bytes for characters like accented letters.

generic

官方文档

https://docs.python.org/3/library/codecs.html#standard-encodings

解决方案

  1. Use Python's `chardet` library to detect encoding and then read with the correct encoding: `import chardet; with open('file.csv', 'rb') as f: result = chardet.detect(f.read()); df = pd.read_csv('file.csv', encoding=result['encoding'])`.
  2. Convert the file to UTF-8 using iconv: `iconv -f WINDOWS-1252 -t UTF-8 input.csv > output.csv`.

无效尝试

常见但无效的做法:

  1. 90% 失败

    Adding UTF-8 BOM to the file does not fix the encoding; it only helps with byte order marking but does not convert the actual character encoding.

  2. 80% 失败

    Manually replacing garbled characters one by one is error-prone and does not scale for large files.