KAFKA-ERR-100 data config_error ai_generated true

Kafka JSON deserialization fails with 'Unknown schema ID' when schema registry is not configured

ID: data/kafka-json-deserialization-with-schema-registry

Also available as: JSON · Markdown · 中文
95%Fix Rate
87%Confidence
1Evidence
2023-04-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Confluent Platform 7.4 active — — —
kafka-python 2.0.2 active — — —
Apache Kafka 3.5 active — — —

Root Cause

Kafka producer writes JSON data with schema registry enabled, but the consumer does not have schema registry configured or the schema ID is not present in the registry.

generic

中文

Kafka生产者启用了模式注册表写入JSON数据，但消费者未配置模式注册表，或者模式ID不在注册表中。

Official Documentation

https://docs.confluent.io/platform/current/schema-registry/index.html

Workarounds

  1. 95% success Configure the consumer to use the same schema registry URL: Add 'schema.registry.url=http://localhost:8081' to consumer properties. 
    Configure the consumer to use the same schema registry URL: Add 'schema.registry.url=http://localhost:8081' to consumer properties.
  2. 80% success If schema registry is unavailable, use a custom deserializer that ignores schema IDs: kafka.deserializers.string_deserializer for raw JSON. 
    If schema registry is unavailable, use a custom deserializer that ignores schema IDs: kafka.deserializers.string_deserializer for raw JSON.

中文步骤

  1. 配置消费者使用相同的模式注册表URL：在消费者属性中添加'schema.registry.url=http://localhost:8081'。
  2. 如果模式注册表不可用，使用忽略模式ID的自定义反序列化器：使用kafka.deserializers.string_deserializer处理原始JSON。

Dead Ends

Common approaches that don't work:

  1. Disabling schema registry on the producer side 80% fail

    Producer may have already written messages with schema IDs; disabling registry only affects new messages, not existing ones.

  2. Manually editing the Kafka topic to remove schema IDs from message headers 90% fail

    Editing message headers is complex and risky; schema IDs are embedded in the message payload, not just headers.