TIMESTAMP_EPOCH_UNIT_MISMATCH data type_error ai_generated true

Timestamp deserialization fails because epoch is in milliseconds but expected in seconds

ID: data/timestamp-epoch-millis-vs-seconds

Also available as: JSON · Markdown · 中文
90%Fix Rate
89%Confidence
1Evidence
2024-06-22First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Apache Kafka 3.5.0 active — — —
Python datetime 3.11 active — — —
Java 17 active — — —

Root Cause

The data source provides timestamps as Unix epoch milliseconds (e.g., 1700000000000) but the deserializer expects seconds (e.g., 1700000000), causing overflow or incorrect dates.

generic

中文

数据源提供Unix纪元毫秒的时间戳（如1700000000000），但反序列化器期望秒（如1700000000），导致溢出或错误日期。

Official Documentation

https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html

Workarounds

  1. 95% success Detect the unit by checking the magnitude: if timestamp > 1e12, treat as milliseconds and divide by 1000. Example in Python: `if ts > 1e12: ts /= 1000`. 
    Detect the unit by checking the magnitude: if timestamp > 1e12, treat as milliseconds and divide by 1000. Example in Python: `if ts > 1e12: ts /= 1000`.
  2. 85% success Configure the deserializer to expect milliseconds explicitly, e.g., in Kafka Connect: `timestamp.converter=org.apache.kafka.connect.json.JsonConverter` with `converter.type=timestamp` and `converter.format=epoch.millis`. 
    Configure the deserializer to expect milliseconds explicitly, e.g., in Kafka Connect: `timestamp.converter=org.apache.kafka.connect.json.JsonConverter` with `converter.type=timestamp` and `converter.format=epoch.millis`.

中文步骤

  1. Detect the unit by checking the magnitude: if timestamp > 1e12, treat as milliseconds and divide by 1000. Example in Python: `if ts > 1e12: ts /= 1000`.
  2. Configure the deserializer to expect milliseconds explicitly, e.g., in Kafka Connect: `timestamp.converter=org.apache.kafka.connect.json.JsonConverter` with `converter.type=timestamp` and `converter.format=epoch.millis`.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Dividing by 1000 in code may cause integer overflow if the timestamp is stored as a 32-bit integer.

  2. 80% fail

    Assuming all timestamps are in milliseconds may break when some sources use seconds.