IOException data data_error ai_generated true

DuckDB Parquet read error: column count mismatch between schema and data pages

ID: data/duckdb-parquet-mismatch-column-count

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
DuckDB 0.9.0+ active — — —
Apache Parquet 1.12.0+ active — — —
PyArrow 12.0.0+ active — — —

Root Cause

Parquet file has a corrupted metadata footer where the schema declares more columns than exist in the actual data pages, often caused by incomplete writes or file truncation.

generic

中文

Parquet 文件的元数据页脚损坏，schema 声明的列数多于实际数据页中的列数，通常由不完整的写入或文件截断引起。

Official Documentation

https://duckdb.org/docs/data/parquet/overview.html

Workarounds

  1. 90% success Repair the file by reading with PyArrow and rewriting: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')` 
    Repair the file by reading with PyArrow and rewriting: `import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`
  2. 70% success Use DuckDB's `read_parquet` with `union_by_name=true` to attempt schema reconciliation: `SELECT * FROM read_parquet('file.parquet', union_by_name=true)` 
    Use DuckDB's `read_parquet` with `union_by_name=true` to attempt schema reconciliation: `SELECT * FROM read_parquet('file.parquet', union_by_name=true)`
  3. 50% success Manually inspect the Parquet metadata with `parquet-tools meta file.parquet` and truncate the schema in the footer if possible. 
    Manually inspect the Parquet metadata with `parquet-tools meta file.parquet` and truncate the schema in the footer if possible.

中文步骤

  1. 使用 PyArrow 读取并重写文件修复：`import pyarrow.parquet as pq; table = pq.read_table('corrupted.parquet'); pq.write_table(table, 'fixed.parquet')`
  2. 使用 DuckDB 的 `read_parquet` 并设置 `union_by_name=true` 尝试 schema 协调：`SELECT * FROM read_parquet('file.parquet', union_by_name=true)`
  3. 使用 `parquet-tools meta file.parquet` 手动检查 Parquet 元数据，并在可能的情况下截断页脚中的 schema。

Dead Ends

Common approaches that don't work:

  1. 60% fail

    The corruption is in the file itself; re-downloading the same source without verification repeats the issue.

  2. 90% fail

    DuckDB does not have an ignore_errors option for Parquet; it will fail on metadata parsing.