# Parquet timestamp values differ by hours after reading with different timezone settings in Spark or PyArrow

- **ID:** `data/parquet-timestamp-timezone-mismatch`
- **Domain:** data
- **Category:** data_error
- **Verification:** ai_generated
- **Fix Rate:** 85%

## Root Cause

Parquet timestamps are stored as UTC by convention, but the reader (e.g., Spark, PyArrow) may apply a local timezone conversion if the session timezone is not set to UTC, leading to off-by-hours errors.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| Spark 3.4 | active | — | — |
| PyArrow 12.0 | active | — | — |
| Pandas 2.0 | active | — | — |

## Workarounds

1. **Set Spark session timezone to UTC before reading: spark.conf.set('spark.sql.session.timeZone', 'UTC').** (95% success)
   ```
   Set Spark session timezone to UTC before reading: spark.conf.set('spark.sql.session.timeZone', 'UTC').
   ```
2. **In PyArrow, specify the timestamp resolution: pq.read_table('file.parquet', timestamp_as_object=True) and manually convert to UTC.** (80% success)
   ```
   In PyArrow, specify the timestamp resolution: pq.read_table('file.parquet', timestamp_as_object=True) and manually convert to UTC.
   ```

## Dead Ends

- **Manually adding or subtracting hours to timestamps after reading** — Hard-coded offsets are fragile and do not account for daylight saving time or varying timezone configurations. (80% fail)
- **Converting all timestamps to local time in the source system before writing** — This breaks the UTC convention and causes inconsistent behavior across different readers. (90% fail)