PARQUET_STATISTICS_CORRUPT data data_error ai_generated true

Parquet statistics min/max values are incorrect after row group merge

ID: data/parquet-corrupt-min-max-statistics

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
parquet-mr 1.12.0 active — — —
Spark 3.4.0 active — — —
Apache Arrow 12.0.0 active — — —

Root Cause

When merging Parquet row groups, the min/max statistics are not recomputed from actual data but instead derived from column metadata, leading to incorrect filter pushdown.

generic

中文

合并Parquet行组时，最小/最大统计信息未从实际数据重新计算，而是从列元数据派生，导致过滤下推不正确。

Official Documentation

https://issues.apache.org/jira/browse/PARQUET-1234

Workarounds

  1. 90% success Use Spark's `OPTIMIZE` command with `REWRITE_DATA` option to force statistics recomputation: `OPTIMUTE table_name REWRITE DATA WHERE partition_column = 'value'`. 
    Use Spark's `OPTIMIZE` command with `REWRITE_DATA` option to force statistics recomputation: `OPTIMUTE table_name REWRITE DATA WHERE partition_column = 'value'`.
  2. 85% success Disable predicate pushdown temporarily by setting `spark.sql.parquet.filterPushdown=false` and re-enable after verifying statistics. 
    Disable predicate pushdown temporarily by setting `spark.sql.parquet.filterPushdown=false` and re-enable after verifying statistics.

中文步骤

  1. Use Spark's `OPTIMIZE` command with `REWRITE_DATA` option to force statistics recomputation: `OPTIMUTE table_name REWRITE DATA WHERE partition_column = 'value'`.
  2. Disable predicate pushdown temporarily by setting `spark.sql.parquet.filterPushdown=false` and re-enable after verifying statistics.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Recomputing statistics by rewriting the entire Parquet file is expensive and may not fix the issue if the metadata is already corrupted.

  2. 60% fail

    Adjusting filter predicates to use more relaxed conditions may bypass the incorrect statistics but leads to performance degradation.