# CSV file with UTF-16 BOM is misinterpreted as UTF-8, producing garbled column headers

- **ID:** `data/csv-encoding-utf16-vs-utf8-bom`
- **Domain:** data
- **Category:** encoding_error
- **Verification:** ai_generated
- **Fix Rate:** 90%

## Root Cause

CSV files saved with UTF-16 encoding (often by Excel on Windows) start with a UTF-16 BOM (0xFFFE) that UTF-8 parsers interpret as two characters, corrupting the first column name.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| Microsoft Excel 365 | active | — | — |
| Python csv module 3.10+ | active | — | — |
| Pandas 1.5.0+ | active | — | — |
| Apache Commons CSV 1.10.0+ | active | — | — |

## Workarounds

1. **Detect and decode UTF-16 BOM explicitly: `with open('file.csv', 'rb') as f: raw = f.read(); if raw[:2] == b'\xff\xfe': text = raw.decode('utf-16'); else: text = raw.decode('utf-8')`** (95% success)
   ```
   Detect and decode UTF-16 BOM explicitly: `with open('file.csv', 'rb') as f: raw = f.read(); if raw[:2] == b'\xff\xfe': text = raw.decode('utf-16'); else: text = raw.decode('utf-8')`
   ```
2. **Use pandas with explicit encoding detection: `import chardet; with open('file.csv', 'rb') as f: enc = chardet.detect(f.read(10000)); df = pd.read_csv('file.csv', encoding=enc['encoding'])`** (90% success)
   ```
   Use pandas with explicit encoding detection: `import chardet; with open('file.csv', 'rb') as f: enc = chardet.detect(f.read(10000)); df = pd.read_csv('file.csv', encoding=enc['encoding'])`
   ```
3. **Convert the file using `iconv -f UTF-16 -t UTF-8 file.csv > fixed.csv`** (85% success)
   ```
   Convert the file using `iconv -f UTF-16 -t UTF-8 file.csv > fixed.csv`
   ```

## Dead Ends

- **** — Notepad may silently change line endings or add extra characters, causing further corruption. (50% fail)
- **** — utf-8-sig only handles UTF-8 BOM (0xEFBBBF), not UTF-16 BOM (0xFFFE). (80% fail)