{
  "id": "data/kafka-connect-avro-schema-evolution-field-removal",
  "signature": "Kafka Connect Avro deserialization fails: field removed in new schema version without default",
  "signature_zh": "Kafka Connect Avro 反序列化失败：新 schema 版本中删除了没有默认值的字段",
  "regex": "SchemaCompatibilityException.*field.*removed.*no default|Avro.*schema.*incompatible.*removed",
  "domain": "data",
  "category": "schema_error",
  "subcategory": null,
  "root_cause": "Avro schema evolution allows adding fields with defaults but not removing fields; a schema that removes a previously existing field without providing a default value breaks backward compatibility.",
  "root_cause_type": "generic",
  "root_cause_zh": "Avro schema 演变允许添加带默认值的字段，但不允许删除字段；删除之前存在的字段而不提供默认值会破坏向后兼容性。",
  "versions": [
    {
      "version": "Confluent Schema Registry 7.4+",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "Apache Avro 1.11.0+",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "Kafka Connect 3.4+",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    }
  ],
  "os_specific": {},
  "dead_ends": [
    {
      "action": "",
      "why_fails": "Breaks all future schema validation and can cause data loss for downstream consumers.",
      "fail_rate": 0.7,
      "condition": "",
      "sources": []
    },
    {
      "action": "",
      "why_fails": "Schema Registry enforces compatibility based on schema ID; editing schema after registration may conflict with existing versions.",
      "fail_rate": 0.5,
      "condition": "",
      "sources": []
    }
  ],
  "workarounds": [
    {
      "action": "Register a new schema version with the removed field re-added as optional with null default: `{\"name\": \"removedField\", \"type\": [\"null\", \"string\"], \"default\": null}`",
      "success_rate": 0.95,
      "how": "Register a new schema version with the removed field re-added as optional with null default: `{\"name\": \"removedField\", \"type\": [\"null\", \"string\"], \"default\": null}`",
      "condition": "",
      "sources": []
    },
    {
      "action": "Use schema evolution with FORWARD compatibility mode if possible, allowing removal but requiring defaults for new fields.",
      "success_rate": 0.8,
      "how": "Use schema evolution with FORWARD compatibility mode if possible, allowing removal but requiring defaults for new fields.",
      "condition": "",
      "sources": []
    },
    {
      "action": "Implement custom deserializer that uses a reader schema with the removed field as optional.",
      "success_rate": 0.85,
      "how": "Implement custom deserializer that uses a reader schema with the removed field as optional.",
      "condition": "",
      "sources": []
    }
  ],
  "workarounds_zh": [
    "注册新 schema 版本，将删除的字段重新添加为可选并带 null 默认值：`{\"name\": \"removedField\", \"type\": [\"null\", \"string\"], \"default\": null}`",
    "如果可能，使用 FORWARD 兼容性模式的 schema 演变，允许删除但要求新字段有默认值。",
    "实现自定义反序列化器，使用包含可选已删除字段的 reader schema。"
  ],
  "transition_graph": {
    "leads_to": [],
    "preceded_by": [],
    "frequently_confused_with": []
  },
  "official_doc_url": "https://docs.confluent.io/platform/current/schema-registry/avro.html#backward-compatibility",
  "official_doc_section": null,
  "error_code": "SchemaCompatibilityException",
  "verification_tier": "ai_generated",
  "confidence": 0.88,
  "fix_success_rate": 0.9,
  "resolvable": "true",
  "first_seen": "2023-11-20",
  "last_confirmed": "2024-06-01",
  "last_updated": "2024-06-01",
  "evidence_count": 1,
  "tags": [],
  "locale": "en",
  "aliases": []
}