data data_error ai_generated partial

Parquet时间戳值在Spark或PyArrow中因不同时区设置读取后相差数小时

Parquet timestamp values differ by hours after reading with different timezone settings in Spark or PyArrow

ID: data/parquet-timestamp-timezone-mismatch

其他格式: JSON · Markdown 中文 · English
85%修复率
86%置信度
1证据数
2023-05-10首次发现

版本兼容性

版本状态引入弃用备注
Spark 3.4 active — — —
PyArrow 12.0 active — — —
Pandas 2.0 active — — —

根因分析

Parquet时间戳按约定存储为UTC，但如果会话时区未设置为UTC，读取器（如Spark、PyArrow）可能会应用本地时区转换，导致数小时的误差。

English

Parquet timestamps are stored as UTC by convention, but the reader (e.g., Spark, PyArrow) may apply a local timezone conversion if the session timezone is not set to UTC, leading to off-by-hours errors.

generic

官方文档

https://spark.apache.org/docs/latest/sql-ref-datetime.html#timestamp-and-timezone

解决方案

  1. 在读取前将Spark会话时区设置为UTC：spark.conf.set('spark.sql.session.timeZone', 'UTC')。
  2. 在PyArrow中，指定时间戳分辨率：pq.read_table('file.parquet', timestamp_as_object=True)并手动转换为UTC。

无效尝试

常见但无效的做法:

  1. Manually adding or subtracting hours to timestamps after reading 80% 失败

    Hard-coded offsets are fragile and do not account for daylight saving time or varying timezone configurations.

  2. Converting all timestamps to local time in the source system before writing 90% 失败

    This breaks the UTC convention and causes inconsistent behavior across different readers.