# CSV file with UTF-8 BOM causes silent data corruption in Excel on Windows

- **ID:** `data/csv-encoding-utf8-with-bom-silent-corruption`
- **Domain:** data
- **Category:** encoding_error
- **Verification:** ai_generated
- **Fix Rate:** 92%

## Root Cause

Excel on Windows interprets BOM-less UTF-8 CSV files as ANSI (Windows-1252), corrupting non-ASCII characters. Adding BOM fixes encoding detection but may cause issues with other tools that don't expect BOM.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| Excel 2019 | active | — | — |
| Excel 365 | active | — | — |
| Excel 2021 | active | — | — |

## Workarounds

1. **Add UTF-8 BOM to CSV files before opening in Excel. In Python: with open('output.csv', 'w', encoding='utf-8-sig') as f: writer = csv.writer(f); writer.writerows(data). The 'utf-8-sig' encoding adds BOM automatically. In command line: sed '1s/^/\xef\xbb\xbf/' input.csv > output.csv** (95% success)
   ```
   Add UTF-8 BOM to CSV files before opening in Excel. In Python: with open('output.csv', 'w', encoding='utf-8-sig') as f: writer = csv.writer(f); writer.writerows(data). The 'utf-8-sig' encoding adds BOM automatically. In command line: sed '1s/^/\xef\xbb\xbf/' input.csv > output.csv
   ```
2. **Use Excel's 'Get Data from Text/CSV' feature instead of double-clicking: Data tab > Get Data > From File > From Text/CSV. Then choose UTF-8 encoding explicitly in the import wizard.** (90% success)
   ```
   Use Excel's 'Get Data from Text/CSV' feature instead of double-clicking: Data tab > Get Data > From File > From Text/CSV. Then choose UTF-8 encoding explicitly in the import wizard.
   ```

## Dead Ends

- **** — This option adds BOM but also changes the file format slightly (e.g., quoting rules), and the file may not be re-importable correctly. (55% fail)
- **** — UTF-16 is not widely supported by CSV parsers and will cause issues with most data processing tools. It also doubles file size. (80% fail)