data type_error ai_generated partial

Parquet浮点数列的值在往返过程中由于模式中的浮点舍入导致0.1 + 0.2 != 0.3

Parquet float column values 0.1 + 0.2 != 0.3 after round-trip due to floating-point rounding in schema

ID: data/parquet-float-precision-loss-rounding

其他格式: JSON · Markdown 中文 · English
75%修复率
82%置信度
1证据数
2023-01-10首次发现

版本兼容性

版本状态引入弃用备注
PyArrow 12.0 active — — —
Spark 3.4 active — — —
Parquet 2.0 active — — —

根因分析

Parquet使用FLOAT（32位）或DOUBLE（64位）类型；写入浮点值时，二进制表示导致精度损失，重新读取时可能在聚合或比较中引入额外的舍入误差。

English

Parquet uses FLOAT (32-bit) or DOUBLE (64-bit) types; when writing float values, binary representation causes precision loss, and reading back may introduce additional rounding errors in aggregation or comparison.

generic

解决方案

  1. 对于高精度列使用DOUBLE而非FLOAT：pd.read_parquet('file.parquet', dtype_backend='pyarrow')并转换为float64。
  2. 在比较浮点值时应用容差：if abs(a - b) < 1e-9: 视为相等。

无效尝试

常见但无效的做法:

  1. Using Decimal type in Parquet schema to store all floats 60% 失败

    Decimal type can reduce precision loss but introduces performance overhead and may not be supported by all readers.

  2. Rounding values to a fixed number of decimal places before writing 70% 失败

    Rounding can mask the issue but does not eliminate floating-point errors; comparisons still fail for edge cases.