AVRO_SCHEMA_EVOLUTION_TYPE_MISMATCH data schema_error ai_generated partial

Avro schema evolution fails when field type changes from int to long

ID: data/avro-schema-evolution-field-type-change

Also available as: JSON · Markdown · 中文
75%Fix Rate
88%Confidence
1Evidence
2023-08-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Apache Avro 1.11.0 active — — —
Confluent Schema Registry 7.5.0 active — — —

Root Cause

Avro schema evolution does not allow changing a field's type from int to long without a custom conversion, as it violates backward compatibility rules.

generic

中文

Avro模式演变不允许将字段类型从int改为long而没有自定义转换，因为这违反了向后兼容性规则。

Official Documentation

https://avro.apache.org/docs/current/spec.html#Schema+Evolution

Workarounds

  1. 85% success Define a new field with the target type (e.g., 'new_field' as long) and deprecate the old field. Update producers to write to the new field and consumers to read from both. 
    Define a new field with the target type (e.g., 'new_field' as long) and deprecate the old field. Update producers to write to the new field and consumers to read from both.
  2. 70% success Use a custom logical type conversion in the schema registry to handle int-to-long promotion, e.g., by setting `avro.schema.conversion` property. 
    Use a custom logical type conversion in the schema registry to handle int-to-long promotion, e.g., by setting `avro.schema.conversion` property.

中文步骤

  1. Define a new field with the target type (e.g., 'new_field' as long) and deprecate the old field. Update producers to write to the new field and consumers to read from both.
  2. Use a custom logical type conversion in the schema registry to handle int-to-long promotion, e.g., by setting `avro.schema.conversion` property.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Simply adding a default value to the field does not resolve the type change issue because Avro checks the type compatibility first.

  2. 90% fail

    Deleting and recreating the schema entirely breaks existing data streams and requires reprocessing all historical data.