由于朱利安日期转换错误，Parquet INT96时间戳读取为5000年以上
Parquet INT96 timestamp reads as year 5000+ due to Julian date conversion error
ID: data/parquet-int96-timestamp-millennium-bug
版本兼容性
| 版本 | 状态 | 引入 | 弃用 | 备注 |
|---|---|---|---|---|
| Apache Parquet 1.12.0 | active | — | — | — |
| Apache Hive 3.1.3 | active | — | — | — |
| Apache Impala 4.0.0 | active | — | — | — |
| pyarrow 13.0.0 | active | — | — | — |
根因分析
Parquet INT96时间戳存储朱利安日数（自公元前4713年以来的天数）和一天中的时间；某些读取器（如旧版Hive、Impala）错误地将朱利安日期解释为Unix纪元偏移，导致日期偏差数百年。
English
Parquet INT96 timestamps store a Julian day number (days since 4713 BC) and time of day; some readers (e.g., older Hive, Impala) incorrectly interpret the Julian date as a Unix epoch offset, causing dates to be centuries off.
官方文档
https://parquet.apache.org/docs/file-format/types/解决方案
-
In pyarrow, read with `pq.read_table(path, use_legacy_int96_timestamps=False)` to use the corrected conversion. Example: `import pyarrow.parquet as pq; table = pq.read_table('data.parquet', use_legacy_int96_timestamps=False)` -
In Spark, set `spark.sql.parquet.int96TimestampConversion.enabled` to `false` and `spark.sql.parquet.int96RebaseModeInRead` to `CORRECTED` to fix the conversion
-
Rewrite the Parquet file using a modern writer (e.g., Spark 3.x) that stores timestamps as INT64 millis instead of INT96, then read with the new file
无效尝试
常见但无效的做法:
-
95% 失败
The CAST operation uses the same broken conversion logic; it will produce the same erroneous future dates.
-
80% 失败
Converting INT96 to STRING often results in a binary representation (e.g., '\x00...') that is not human-readable and cannot be parsed into a date.
-
60% 失败
The bug is in the INT96 conversion logic, which may still be present in newer versions if the file was written by a different tool (e.g., Spark) that uses a non-standard INT96 encoding.