kafka config_error ai_generated true

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

ID: kafka/max-message-bytes-exceeded

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-01-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 2.8.0 active — — —
Kafka 3.0.0 active — — —
Kafka 3.4.0 active — — —
Kafka 3.6.0 active — — —

Root Cause

A producer sent a message that exceeds the broker's max.message.bytes or the topic's max.message.bytes configuration, causing the broker to reject it.

generic

中文

生产者发送的消息超过了代理的 max.message.bytes 或主题的 max.message.bytes 配置，导致代理拒绝该消息。

Official Documentation

https://kafka.apache.org/documentation/#max.message.bytes

Workarounds

  1. 90% success Increase the broker's max.message.bytes in server.properties and the topic's max.message.bytes if overridden: # server.properties max.message.bytes=10485760 # 10 MB # Topic override alter topic my_topic --config max.message.bytes=10485760 Also increase the producer's max.request.size to match: Properties props = new Properties(); props.put("max.request.size", 10485760); 
    Increase the broker's max.message.bytes in server.properties and the topic's max.message.bytes if overridden:

# server.properties
max.message.bytes=10485760  # 10 MB

# Topic override
alter topic my_topic --config max.message.bytes=10485760

Also increase the producer's max.request.size to match:
Properties props = new Properties();
props.put("max.request.size", 10485760);
  2. 85% success Split large messages into smaller chunks at the application level before sending, and reassemble on the consumer side. For example, use a message chunking library or implement custom logic. 
    Split large messages into smaller chunks at the application level before sending, and reassemble on the consumer side. For example, use a message chunking library or implement custom logic.

中文步骤

  1. Increase the broker's max.message.bytes in server.properties and the topic's max.message.bytes if overridden:

# server.properties
max.message.bytes=10485760  # 10 MB

# Topic override
alter topic my_topic --config max.message.bytes=10485760

Also increase the producer's max.request.size to match:
Properties props = new Properties();
props.put("max.request.size", 10485760);
  2. Split large messages into smaller chunks at the application level before sending, and reassemble on the consumer side. For example, use a message chunking library or implement custom logic.

Dead Ends

Common approaches that don't work:

  1. Increase the producer's max.request.size only 80% fail

    max.request.size controls the client-side buffer, but the broker's max.message.bytes still blocks the message; both must be increased.

  2. Set the message compression to gzip 70% fail

    Compression reduces size but does not guarantee the compressed message will be under the limit; large uncompressed data may still exceed it.