kafka runtime_error ai_generated true

org.apache.kafka.common.errors.RecordBatchTooLargeException: The request included a batch of records larger than the maximum allowed size

ID: kafka/record-batch-too-large

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2024-06-18First Seen

Root Cause

A single batch of records exceeds the broker's max.message.bytes or message.max.bytes configuration.

generic

中文

单个记录批次大小超过代理的 max.message.bytes 或 message.max.bytes 配置。

Official Documentation

https://kafka.apache.org/documentation/#brokerconfigs_message.max.bytes

Workarounds

  1. 90% success Increase max.message.bytes on the broker: kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config 'message.max.bytes=20971520' 
    Increase max.message.bytes on the broker: kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config 'message.max.bytes=20971520'
  2. 80% success Reduce batch.size on the producer to limit batch size: props.put("batch.size", "16384"); 
    Reduce batch.size on the producer to limit batch size: props.put("batch.size", "16384");
  3. 85% success Split large records into smaller ones before sending to Kafka. 
    Split large records into smaller ones before sending to Kafka.

中文步骤

  1. 在代理上增加 max.message.bytes：kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config 'message.max.bytes=20971520'
  2. 在生产者上减少 batch.size 以限制批次大小：props.put("batch.size", "16384");
  3. 在发送到Kafka之前将大记录拆分为较小的记录。

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Increasing max.request.size on the producer without also increasing broker limits will still cause rejection.

  2. 70% fail

    Compressing the batch may not reduce size enough if individual records are too large.