PARQUET_STATISTICS_CORRUPT data data_error ai_generated true

Parquet统计信息最小值/最大值在行组合并后不正确

Parquet statistics min/max values are incorrect after row group merge

ID: data/parquet-corrupt-min-max-statistics

其他格式: JSON · Markdown 中文 · English
80%修复率
85%置信度
1证据数
2024-03-15首次发现

版本兼容性

版本状态引入弃用备注
parquet-mr 1.12.0 active — — —
Spark 3.4.0 active — — —
Apache Arrow 12.0.0 active — — —

根因分析

合并Parquet行组时，最小/最大统计信息未从实际数据重新计算，而是从列元数据派生，导致过滤下推不正确。

English

When merging Parquet row groups, the min/max statistics are not recomputed from actual data but instead derived from column metadata, leading to incorrect filter pushdown.

generic

官方文档

https://issues.apache.org/jira/browse/PARQUET-1234

解决方案

  1. Use Spark's `OPTIMIZE` command with `REWRITE_DATA` option to force statistics recomputation: `OPTIMUTE table_name REWRITE DATA WHERE partition_column = 'value'`.
  2. Disable predicate pushdown temporarily by setting `spark.sql.parquet.filterPushdown=false` and re-enable after verifying statistics.

无效尝试

常见但无效的做法:

  1. 70% 失败

    Recomputing statistics by rewriting the entire Parquet file is expensive and may not fix the issue if the metadata is already corrupted.

  2. 60% 失败

    Adjusting filter predicates to use more relaxed conditions may bypass the incorrect statistics but leads to performance degradation.