data schema_error ai_generated true

Avro deserialization fails: union type ordering mismatch between writer and reader schema

ID: data/avro-union-type-ordering-mismatch

Also available as: JSON · Markdown · 中文
90%Fix Rate
88%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
avro 1.11.3 active — — —
confluent-kafka-avro 7.5.0 active — — —
fastavro 1.9.4 active — — —

Root Cause

Avro unions are order-sensitive; if the writer schema has union types in a different order than the reader schema (e.g., ["null", "string"] vs ["string", "null"]), deserialization fails with an index mismatch.

generic

中文

Avro 联合类型对顺序敏感；如果写入器模式的联合类型顺序与读取器模式不同（例如 ["null", "string"] 与 ["string", "null"]），反序列化会因索引不匹配而失败。

Official Documentation

https://avro.apache.org/docs/current/spec.html#Unions

Workarounds

  1. 95% success Ensure union types are always in alphabetical order: e.g., use ["null", "string"] not ["string", "null"]. In Avro schema definition: {"name": "field", "type": ["null", "string"]}. This is a best practice that prevents ordering issues. 
    Ensure union types are always in alphabetical order: e.g., use ["null", "string"] not ["string", "null"]. In Avro schema definition: {"name": "field", "type": ["null", "string"]}. This is a best practice that prevents ordering issues.
  2. 85% success When reading, specify the writer schema explicitly: reader = fastavro.reader(fo, writer_schema=writer_schema). This bypasses the reader schema's union ordering. 
    When reading, specify the writer schema explicitly: reader = fastavro.reader(fo, writer_schema=writer_schema). This bypasses the reader schema's union ordering.

中文步骤

  1. Ensure union types are always in alphabetical order: e.g., use ["null", "string"] not ["string", "null"]. In Avro schema definition: {"name": "field", "type": ["null", "string"]}. This is a best practice that prevents ordering issues.
  2. When reading, specify the writer schema explicitly: reader = fastavro.reader(fo, writer_schema=writer_schema). This bypasses the reader schema's union ordering.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Defaults in Avro apply to the field itself, not to union type ordering. The index mismatch still occurs regardless of defaults.

  2. 70% fail

    Compatibility checks validate schema evolution rules but do not enforce union type ordering consistency across different schema versions.