data type_error ai_generated true

Avro deserialization fails: union field expects a specific branch but got null

ID: data/avro-record-schema-union-mismatch

Also available as: JSON · Markdown · 中文
85%Fix Rate
84%Confidence
1Evidence
2023-02-14First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Apache Avro 1.11.1 active — — —
Confluent Avro Serializer 7.4 active — — —
Python avro 1.11.0 active — — —

Root Cause

An Avro schema defines a union field (e.g., ["null", "string"]) but the data contains a null value when the schema expects a non-null branch, or vice versa, due to schema evolution or data corruption.

generic

中文

Avro模式定义了一个联合字段（例如，["null", "string"]），但由于模式演变或数据损坏，数据包含null值而模式期望非null分支，或相反。

Official Documentation

https://avro.apache.org/docs/current/spec.html#Unions

Workarounds

  1. 90% success Update the Avro schema to include null as the first branch in the union: change '["string"]' to '["null", "string"]' and set default to null. 
    Update the Avro schema to include null as the first branch in the union: change '["string"]' to '["null", "string"]' and set default to null.
  2. 80% success Use a custom deserializer that gracefully handles null values by mapping them to a default string: if val is None: val = 'default'. 
    Use a custom deserializer that gracefully handles null values by mapping them to a default string: if val is None: val = 'default'.

中文步骤

  1. 更新Avro模式，将null作为联合的第一个分支：将'["string"]'改为'["null", "string"]'并将默认值设为null。
  2. 使用自定义反序列化器优雅地处理null值，将其映射为默认字符串：if val is None: val = 'default'。

Dead Ends

Common approaches that don't work:

  1. Adding a default value of null to the union field in the schema 70% fail

    The default value applies during schema evolution but does not fix existing data that has mismatched branches.

  2. Changing the entire field type to a single type (e.g., string) to avoid unions 90% fail

    This breaks compatibility with existing data and requires re-encoding all messages.