kafka config_error ai_generated true

org.apache.kafka.common.errors.RecordTooLargeException: 请求包含的消息大于服务器将接受的最大消息大小。

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

ID: kafka/max-message-bytes-exceeded

其他格式: JSON · Markdown 中文 · English
85%修复率
88%置信度
1证据数
2023-01-20首次发现

版本兼容性

版本状态引入弃用备注
Kafka 2.8.0 active — — —
Kafka 3.0.0 active — — —
Kafka 3.4.0 active — — —
Kafka 3.6.0 active — — —

根因分析

生产者发送的消息超过了代理的 max.message.bytes 或主题的 max.message.bytes 配置，导致代理拒绝该消息。

English

A producer sent a message that exceeds the broker's max.message.bytes or the topic's max.message.bytes configuration, causing the broker to reject it.

generic

官方文档

https://kafka.apache.org/documentation/#max.message.bytes

解决方案

  1. Increase the broker's max.message.bytes in server.properties and the topic's max.message.bytes if overridden:

# server.properties
max.message.bytes=10485760  # 10 MB

# Topic override
alter topic my_topic --config max.message.bytes=10485760

Also increase the producer's max.request.size to match:
Properties props = new Properties();
props.put("max.request.size", 10485760);
  2. Split large messages into smaller chunks at the application level before sending, and reassemble on the consumer side. For example, use a message chunking library or implement custom logic.

无效尝试

常见但无效的做法:

  1. Increase the producer's max.request.size only 80% 失败

    max.request.size controls the client-side buffer, but the broker's max.message.bytes still blocks the message; both must be increased.

  2. Set the message compression to gzip 70% 失败

    Compression reduces size but does not guarantee the compressed message will be under the limit; large uncompressed data may still exceed it.