ParquetCorruptStatisticsException data data_error ai_generated true

Parquet corrupted min/max statistics cause predicate pushdown to skip valid rows

ID: data/parquet-corrupted-min-max-statistics

Also available as: JSON · Markdown · 中文
85%Fix Rate
85%Confidence
1Evidence
2023-08-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Parquet 1.12.0+ active — — —
Apache Spark 3.2+ active — — —
Apache Arrow 8.0.0+ active — — —
DuckDB 0.6.0+ active — — —

Root Cause

Parquet file footer stores column chunk min/max statistics that are inconsistent with actual data, causing predicate pushdown filters to incorrectly skip row groups.

generic

中文

Parquet 文件页脚存储的列块最小/最大统计信息与实际数据不一致，导致谓词下推过滤器错误地跳过行组。

Official Documentation

https://issues.apache.org/jira/browse/PARQUET-2210

Workarounds

  1. 90% success Read the Parquet file with statistics disabled: `spark.read.option("parquet.filter.statistics.enabled", "false").parquet("path")` 
    Read the Parquet file with statistics disabled: `spark.read.option("parquet.filter.statistics.enabled", "false").parquet("path")`
  2. 95% success Rewrite the Parquet file using a different library like PyArrow: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')` 
    Rewrite the Parquet file using a different library like PyArrow: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`
  3. 85% success Validate statistics by scanning row groups manually: `pq.ParquetFile('file.parquet').metadata.row_group(0).column(0).statistics` and compare with actual data range. 
    Validate statistics by scanning row groups manually: `pq.ParquetFile('file.parquet').metadata.row_group(0).column(0).statistics` and compare with actual data range.

中文步骤

  1. 禁用统计信息读取 Parquet 文件：`spark.read.option("parquet.filter.statistics.enabled", "false").parquet("path")`
  2. 使用不同库重写 Parquet 文件，如 PyArrow：`import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`
  3. 手动验证统计信息：`pq.ParquetFile('file.parquet').metadata.row_group(0).column(0).statistics` 并与实际数据范围比较。

Dead Ends

Common approaches that don't work:

  1. 40% fail

    Spark uses the same writer logic that may reproduce corrupted statistics if the issue is in the writer library version.

  2. 60% fail

    Overly broad fix that degrades performance for all queries, not just the affected files.