data type_error ai_generated partial

Parquet float column values 0.1 + 0.2 != 0.3 after round-trip due to floating-point rounding in schema

ID: data/parquet-float-precision-loss-rounding

Also available as: JSON · Markdown · 中文
75%Fix Rate
82%Confidence
1Evidence
2023-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
PyArrow 12.0 active — — —
Spark 3.4 active — — —
Parquet 2.0 active — — —

Root Cause

Parquet uses FLOAT (32-bit) or DOUBLE (64-bit) types; when writing float values, binary representation causes precision loss, and reading back may introduce additional rounding errors in aggregation or comparison.

generic

中文

Parquet使用FLOAT（32位）或DOUBLE（64位）类型；写入浮点值时，二进制表示导致精度损失，重新读取时可能在聚合或比较中引入额外的舍入误差。

Workarounds

  1. 80% success Use DOUBLE instead of FLOAT for high-precision columns: pd.read_parquet('file.parquet', dtype_backend='pyarrow') and cast to float64. 
    Use DOUBLE instead of FLOAT for high-precision columns: pd.read_parquet('file.parquet', dtype_backend='pyarrow') and cast to float64.
  2. 90% success Apply a tolerance when comparing float values: if abs(a - b) < 1e-9: treat as equal. 
    Apply a tolerance when comparing float values: if abs(a - b) < 1e-9: treat as equal.

中文步骤

  1. 对于高精度列使用DOUBLE而非FLOAT：pd.read_parquet('file.parquet', dtype_backend='pyarrow')并转换为float64。
  2. 在比较浮点值时应用容差：if abs(a - b) < 1e-9: 视为相等。

Dead Ends

Common approaches that don't work:

  1. Using Decimal type in Parquet schema to store all floats 60% fail

    Decimal type can reduce precision loss but introduces performance overhead and may not be supported by all readers.

  2. Rounding values to a fixed number of decimal places before writing 70% fail

    Rounding can mask the issue but does not eliminate floating-point errors; comparisons still fail for edge cases.