42201 data schema_error ai_generated true

SchemaRegistryException: schema is not backward-compatible — field 'phone_number' added without default in version 3

ID: data/avro-schema-registry-backward-transitive-failure

Also available as: JSON · Markdown · 中文
90%Fix Rate
92%Confidence
1Evidence
2023-11-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Confluent Schema Registry 7.5.0 active — — —
Apache Avro 1.11.3 active — — —
Kafka 3.6.0 active — — —

Root Cause

Avro schema evolution requires new fields to have default values when backward compatibility is enforced; adding a field without a default breaks consumers that read old data.

generic

中文

Avro模式演化要求在启用向后兼容性时新字段必须有默认值；添加没有默认值的字段会破坏读取旧数据的消费者。

Official Documentation

https://docs.confluent.io/platform/current/schema-registry/avro.html

Workarounds

  1. 95% success Update the Avro schema to add the field with a default value: `{"name": "phone_number", "type": ["null", "string"], "default": null}` and register as version 4 with backward compatibility 
    Update the Avro schema to add the field with a default value: `{"name": "phone_number", "type": ["null", "string"], "default": null}` and register as version 4 with backward compatibility
  2. 85% success If the field must be required, change compatibility to FORWARD_TRANSITIVE, register the new schema, then update consumers to handle the new field before switching back to BACKWARD 
    If the field must be required, change compatibility to FORWARD_TRANSITIVE, register the new schema, then update consumers to handle the new field before switching back to BACKWARD
  3. 90% success Use Schema Registry REST API to set compatibility to BACKWARD_TRANSITIVE on the subject, then re-register: `curl -X PUT -H "Content-Type: application/json" --data '{"compatibility": "BACKWARD_TRANSITIVE"}' http://localhost:8081/config/<subject>` 
    Use Schema Registry REST API to set compatibility to BACKWARD_TRANSITIVE on the subject, then re-register: `curl -X PUT -H "Content-Type: application/json" --data '{"compatibility": "BACKWARD_TRANSITIVE"}' http://localhost:8081/config/<subject>`

中文步骤

  1. Update the Avro schema to add the field with a default value: `{"name": "phone_number", "type": ["null", "string"], "default": null}` and register as version 4 with backward compatibility
  2. If the field must be required, change compatibility to FORWARD_TRANSITIVE, register the new schema, then update consumers to handle the new field before switching back to BACKWARD
  3. Use Schema Registry REST API to set compatibility to BACKWARD_TRANSITIVE on the subject, then re-register: `curl -X PUT -H "Content-Type: application/json" --data '{"compatibility": "BACKWARD_TRANSITIVE"}' http://localhost:8081/config/<subject>`

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Schema Registry enforces compatibility globally; deleting a version may break existing producers/consumers and cause data loss if they reference that version ID.

  2. 75% fail

    Disabling compatibility entirely removes all checks, allowing any schema change, which can cause silent data corruption for downstream consumers that expect a specific structure.

  3. 100% fail

    The producer will fail to serialize because the schema must match the registered schema exactly (Avro writer schema validation).