SchemaRegistryException data schema_error ai_generated partial

Avro serialization fails with SchemaRegistryException: incompatible schema version

ID: data/avro-schema-registry-serialization-version-mismatch

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Confluent Platform 7.5 active — — —
Confluent Platform 7.6 active — — —
Apache Kafka 3.5 active — — —
Apache Kafka 3.6 active — — —

Root Cause

The Confluent Schema Registry enforces compatibility rules (BACKWARD, FORWARD, FULL) and rejects schema evolution that violates the configured policy, often due to removing fields or changing types in a way that breaks existing consumers.

generic

中文

Confluent Schema Registry强制执行兼容性规则（BACKWARD、FORWARD、FULL），并拒绝违反配置策略的schema演进，通常因为删除字段或更改类型导致破坏现有消费者。

Official Documentation

https://docs.confluent.io/platform/current/schema-registry/avro.html

Workarounds

  1. 85% success Temporarily set compatibility to BACKWARD_TRANSITIVE for the subject, register the new schema, then revert to original policy: `curl -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' -d '{"compatibility": "BACKWARD_TRANSITIVE"}' https://registry:8081/config/subject-name` 
    Temporarily set compatibility to BACKWARD_TRANSITIVE for the subject, register the new schema, then revert to original policy: `curl -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' -d '{"compatibility": "BACKWARD_TRANSITIVE"}' https://registry:8081/config/subject-name`
  2. 90% success Add the new field with a default value to maintain backward compatibility: `{"name": "new_field", "type": "string", "default": ""}` 
    Add the new field with a default value to maintain backward compatibility: `{"name": "new_field", "type": "string", "default": ""}`
  3. 88% success Use FULL_TRANSITIVE compatibility and register a new version that adds fields with defaults and removes no fields 
    Use FULL_TRANSITIVE compatibility and register a new version that adds fields with defaults and removes no fields

中文步骤

  1. Temporarily set compatibility to BACKWARD_TRANSITIVE for the subject, register the new schema, then revert to original policy: `curl -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' -d '{"compatibility": "BACKWARD_TRANSITIVE"}' https://registry:8081/config/subject-name`
  2. Add the new field with a default value to maintain backward compatibility: `{"name": "new_field", "type": "string", "default": ""}`
  3. Use FULL_TRANSITIVE compatibility and register a new version that adds fields with defaults and removes no fields

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Schema Registry does not allow deletion of schemas that have been used; you must use soft delete or wait for the cleanup policy.

  2. 70% fail

    This creates a new subject that existing consumers are not subscribed to, breaking data flow and requiring manual topic reconfiguration.

  3. 60% fail

    Disabling compatibility entirely allows any schema change, which can silently break downstream consumers and cause production incidents.