SchemaCompatibilityException data schema_error ai_generated true

Kafka Connect Avro deserialization fails: field removed in new schema version without default

ID: data/kafka-connect-avro-schema-evolution-field-removal

Also available as: JSON · Markdown · 中文
90%Fix Rate
88%Confidence
1Evidence
2023-11-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Confluent Schema Registry 7.4+ active — — —
Apache Avro 1.11.0+ active — — —
Kafka Connect 3.4+ active — — —

Root Cause

Avro schema evolution allows adding fields with defaults but not removing fields; a schema that removes a previously existing field without providing a default value breaks backward compatibility.

generic

中文

Avro schema 演变允许添加带默认值的字段，但不允许删除字段；删除之前存在的字段而不提供默认值会破坏向后兼容性。

Official Documentation

https://docs.confluent.io/platform/current/schema-registry/avro.html#backward-compatibility

Workarounds

  1. 95% success Register a new schema version with the removed field re-added as optional with null default: `{"name": "removedField", "type": ["null", "string"], "default": null}` 
    Register a new schema version with the removed field re-added as optional with null default: `{"name": "removedField", "type": ["null", "string"], "default": null}`
  2. 80% success Use schema evolution with FORWARD compatibility mode if possible, allowing removal but requiring defaults for new fields. 
    Use schema evolution with FORWARD compatibility mode if possible, allowing removal but requiring defaults for new fields.
  3. 85% success Implement custom deserializer that uses a reader schema with the removed field as optional. 
    Implement custom deserializer that uses a reader schema with the removed field as optional.

中文步骤

  1. 注册新 schema 版本，将删除的字段重新添加为可选并带 null 默认值：`{"name": "removedField", "type": ["null", "string"], "default": null}`
  2. 如果可能，使用 FORWARD 兼容性模式的 schema 演变，允许删除但要求新字段有默认值。
  3. 实现自定义反序列化器，使用包含可选已删除字段的 reader schema。

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Breaks all future schema validation and can cause data loss for downstream consumers.

  2. 50% fail

    Schema Registry enforces compatibility based on schema ID; editing schema after registration may conflict with existing versions.