kafka resource_error ai_generated true

java.io.IOException: No space left on device while writing to log segment /var/lib/kafka/data/my_topic-0/00000000000000000000.log

ID: kafka/disk-full-log-segment

Also available as: JSON · Markdown · 中文
95%Fix Rate
90%Confidence
1Evidence
2023-12-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 3.5.0 active — — —
Kafka 3.6.1 active — — —

Root Cause

Kafka broker's data directory has run out of disk space, preventing log segment writes and causing broker to become unhealthy.

generic

中文

Kafka代理的数据目录磁盘空间已满，阻止日志段写入并导致代理变得不健康。

Official Documentation

https://kafka.apache.org/documentation/#disk_space

Workarounds

  1. 95% success Run `df -h /var/lib/kafka/data` to confirm disk usage, then add more disk space or move data to a larger partition. Use `log.dirs=/data/kafka` in server.properties to point to a new location. 
    Run `df -h /var/lib/kafka/data` to confirm disk usage, then add more disk space or move data to a larger partition. Use `log.dirs=/data/kafka` in server.properties to point to a new location.
  2. 85% success Trigger log compaction or deletion by reducing `log.retention.hours=24` or `log.retention.bytes=10737418240` (10GB) and restart broker gracefully. 
    Trigger log compaction or deletion by reducing `log.retention.hours=24` or `log.retention.bytes=10737418240` (10GB) and restart broker gracefully.
  3. 90% success Example: `kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic --alter --add-config retention.ms=86400000` to reduce retention to 24 hours. 
    Example: `kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic --alter --add-config retention.ms=86400000` to reduce retention to 24 hours.

中文步骤

  1. Run `df -h /var/lib/kafka/data` to confirm disk usage, then add more disk space or move data to a larger partition. Use `log.dirs=/data/kafka` in server.properties to point to a new location.
  2. Trigger log compaction or deletion by reducing `log.retention.hours=24` or `log.retention.bytes=10737418240` (10GB) and restart broker gracefully.
  3. Example: `kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic --alter --add-config retention.ms=86400000` to reduce retention to 24 hours.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Kafka manages segments internally; manual deletion breaks consistency.

  2. 60% fail

    It may trigger aggressive cleanup but does not free space immediately if cleanup policy is not triggered.