data data_error ai_generated partial

Parquet timestamp values differ by hours after reading with different timezone settings in Spark or PyArrow

ID: data/parquet-timestamp-timezone-mismatch

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2023-05-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Spark 3.4 active — — —
PyArrow 12.0 active — — —
Pandas 2.0 active — — —

Root Cause

Parquet timestamps are stored as UTC by convention, but the reader (e.g., Spark, PyArrow) may apply a local timezone conversion if the session timezone is not set to UTC, leading to off-by-hours errors.

generic

中文

Parquet时间戳按约定存储为UTC，但如果会话时区未设置为UTC，读取器（如Spark、PyArrow）可能会应用本地时区转换，导致数小时的误差。

Official Documentation

https://spark.apache.org/docs/latest/sql-ref-datetime.html#timestamp-and-timezone

Workarounds

  1. 95% success Set Spark session timezone to UTC before reading: spark.conf.set('spark.sql.session.timeZone', 'UTC'). 
    Set Spark session timezone to UTC before reading: spark.conf.set('spark.sql.session.timeZone', 'UTC').
  2. 80% success In PyArrow, specify the timestamp resolution: pq.read_table('file.parquet', timestamp_as_object=True) and manually convert to UTC. 
    In PyArrow, specify the timestamp resolution: pq.read_table('file.parquet', timestamp_as_object=True) and manually convert to UTC.

中文步骤

  1. 在读取前将Spark会话时区设置为UTC：spark.conf.set('spark.sql.session.timeZone', 'UTC')。
  2. 在PyArrow中，指定时间戳分辨率：pq.read_table('file.parquet', timestamp_as_object=True)并手动转换为UTC。

Dead Ends

Common approaches that don't work:

  1. Manually adding or subtracting hours to timestamps after reading 80% fail

    Hard-coded offsets are fragile and do not account for daylight saving time or varying timezone configurations.

  2. Converting all timestamps to local time in the source system before writing 90% fail

    This breaks the UTC convention and causes inconsistent behavior across different readers.