kafka data_error ai_generated true

org.apache.kafka.common.errors.InvalidCommitOffsetSizeException: 提交偏移量大小 1048577 超过了最大允许大小 1048576

org.apache.kafka.common.errors.InvalidCommitOffsetSizeException: The commit offset size 1048577 exceeds the maximum allowed size 1048576

ID: kafka/invalid-commit-offset-size

其他格式: JSON · Markdown 中文 · English
75%修复率
83%置信度
1证据数
2024-02-10首次发现

版本兼容性

版本状态引入弃用备注
kafka_2.13-3.5.0 active — — —
kafka_2.13-3.6.0 active — — —
kafka_2.13-3.7.0 active — — —

根因分析

消费者组偏移量提交元数据超过了配置的最大大小（默认 1 MB），通常是由于在提交请求中存储了大型自定义元数据。

English

Consumer group offset commit metadata exceeds the configured maximum size (default 1 MB), often due to storing large custom metadata in the commit request.

generic

官方文档

https://kafka.apache.org/documentation/#consumerconfigs_offset.commit.metadata.max.size

解决方案

  1. Reduce the size of custom metadata passed to `commitSync(Map<TopicPartition, OffsetAndMetadata>)` by removing large objects or storing references externally. Example: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "small_metadata")));`
  2. Increase `offset.commit.metadata.max.size` in the broker configuration (e.g., to 2097152 bytes) and restart the broker. This setting is dynamic but requires broker restart to take effect.
  3. If metadata is not needed, set it to empty string: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "")));`

无效尝试

常见但无效的做法:

  1. 95% 失败

    This affects replication, not the metadata size limit; the commit size limit is independent.

  2. 85% 失败

    This controls how many records are polled, not the offset metadata size; the error occurs during commit, not poll.

  3. 99% 失败

    This removes all committed offsets for all groups, causing data loss and does not fix the size limit; the topic is recreated with default config.