data data_error ai_generated true

CSV 空值与空字符串歧义——"" 和无值在 pandas 中均变为 None

CSV null vs empty string ambiguity — "" and no-value both become None in pandas

ID: data/csv-null-vs-empty-string-ambiguity

其他格式: JSON · Markdown 中文 · English
82%修复率
85%置信度
1证据数
2023-03-15首次发现

版本兼容性

版本状态引入弃用备注
pandas 1.5.3 active — — —
pandas 2.0.0 active — — —
pandas 2.1.4 active — — —

根因分析

Pandas read_csv 默认将空引号字符串和缺失字段均视为 NaN，丢失了空字符串与空值之间的区别。

English

Pandas read_csv treats both empty quoted strings and missing fields as NaN by default, losing the distinction between empty strings and null values.

generic

官方文档

https://pandas.pydata.org/docs/user_guide/io.html#io-read-csv-table

解决方案

  1. Use pd.read_csv(..., keep_default_na=False, na_values=[''], dtype=str) and then manually convert empty strings to None where needed. Example: df = pd.read_csv('data.csv', keep_default_na=False, na_values=[''], dtype={'col1': str}); df['col1'] = df['col1'].replace('', pd.NA)
  2. Pre-process CSV by replacing empty quoted fields with a sentinel like '__NULL__', then map back after reading: sed 's/""/__NULL__/g' input.csv | pd.read_csv(...); df.replace('__NULL__', pd.NA)

无效尝试

常见但无效的做法:

  1. 65% 失败

    This makes pandas treat no-value cells as empty strings too, but still converts empty quoted strings to NaN.

  2. 70% 失败

    Disables all NA detection, but also prevents legitimate NaN values from being recognized, breaking downstream null handling.