kafka data_error ai_generated true

org.apache.kafka.common.errors.InvalidCommitOffsetSizeException: The commit offset size 1048577 exceeds the maximum allowed size 1048576

ID: kafka/invalid-commit-offset-size

Also available as: JSON · Markdown · 中文
75%Fix Rate
83%Confidence
1Evidence
2024-02-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kafka_2.13-3.5.0 active — — —
kafka_2.13-3.6.0 active — — —
kafka_2.13-3.7.0 active — — —

Root Cause

Consumer group offset commit metadata exceeds the configured maximum size (default 1 MB), often due to storing large custom metadata in the commit request.

generic

中文

消费者组偏移量提交元数据超过了配置的最大大小（默认 1 MB），通常是由于在提交请求中存储了大型自定义元数据。

Official Documentation

https://kafka.apache.org/documentation/#consumerconfigs_offset.commit.metadata.max.size

Workarounds

  1. 90% success Reduce the size of custom metadata passed to `commitSync(Map<TopicPartition, OffsetAndMetadata>)` by removing large objects or storing references externally. Example: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "small_metadata")));` 
    Reduce the size of custom metadata passed to `commitSync(Map<TopicPartition, OffsetAndMetadata>)` by removing large objects or storing references externally. Example: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "small_metadata")));`
  2. 80% success Increase `offset.commit.metadata.max.size` in the broker configuration (e.g., to 2097152 bytes) and restart the broker. This setting is dynamic but requires broker restart to take effect. 
    Increase `offset.commit.metadata.max.size` in the broker configuration (e.g., to 2097152 bytes) and restart the broker. This setting is dynamic but requires broker restart to take effect.
  3. 95% success If metadata is not needed, set it to empty string: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "")));` 
    If metadata is not needed, set it to empty string: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "")));`

中文步骤

  1. Reduce the size of custom metadata passed to `commitSync(Map<TopicPartition, OffsetAndMetadata>)` by removing large objects or storing references externally. Example: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "small_metadata")));`
  2. Increase `offset.commit.metadata.max.size` in the broker configuration (e.g., to 2097152 bytes) and restart the broker. This setting is dynamic but requires broker restart to take effect.
  3. If metadata is not needed, set it to empty string: `consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(offset, "")));`

Dead Ends

Common approaches that don't work:

  1. 95% fail

    This affects replication, not the metadata size limit; the commit size limit is independent.

  2. 85% fail

    This controls how many records are polled, not the offset metadata size; the error occurs during commit, not poll.

  3. 99% fail

    This removes all committed offsets for all groups, causing data loss and does not fix the size limit; the topic is recreated with default config.