# Parquet corrupted min/max statistics cause predicate pushdown to skip valid rows

- **ID:** `data/parquet-corrupted-min-max-statistics`
- **Domain:** data
- **Category:** data_error
- **Error Code:** `ParquetCorruptStatisticsException`
- **Verification:** ai_generated
- **Fix Rate:** 85%

## Root Cause

Parquet file footer stores column chunk min/max statistics that are inconsistent with actual data, causing predicate pushdown filters to incorrectly skip row groups.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| Parquet 1.12.0+ | active | — | — |
| Apache Spark 3.2+ | active | — | — |
| Apache Arrow 8.0.0+ | active | — | — |
| DuckDB 0.6.0+ | active | — | — |

## Workarounds

1. **Read the Parquet file with statistics disabled: `spark.read.option("parquet.filter.statistics.enabled", "false").parquet("path")`** (90% success)
   ```
   Read the Parquet file with statistics disabled: `spark.read.option("parquet.filter.statistics.enabled", "false").parquet("path")`
   ```
2. **Rewrite the Parquet file using a different library like PyArrow: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`** (95% success)
   ```
   Rewrite the Parquet file using a different library like PyArrow: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`
   ```
3. **Validate statistics by scanning row groups manually: `pq.ParquetFile('file.parquet').metadata.row_group(0).column(0).statistics` and compare with actual data range.** (85% success)
   ```
   Validate statistics by scanning row groups manually: `pq.ParquetFile('file.parquet').metadata.row_group(0).column(0).statistics` and compare with actual data range.
   ```

## Dead Ends

- **** — Spark uses the same writer logic that may reproduce corrupted statistics if the issue is in the writer library version. (40% fail)
- **** — Overly broad fix that degrades performance for all queries, not just the affected files. (60% fail)