320 communication connection_error ai_generated true

AMQPConnectionError: connection forced (320) CONNECTION_FORCED

ID: communication/amqp-connection-forced

Also available as: JSON · Markdown · 中文
83%Fix Rate
86%Confidence
1Evidence
2023-09-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
RabbitMQ 3.11 active — — —
RabbitMQ 3.12 active — — —
RabbitMQ 3.13 active — — —

Root Cause

The AMQP broker (e.g., RabbitMQ) forcibly closed the connection due to a resource alarm (memory or disk) or an internal error, often signaled by the connection.blocked method.

generic

中文

AMQP 代理（如 RabbitMQ）因资源告警（内存或磁盘）或内部错误强制关闭连接，通常由 connection.blocked 方法指示。

Official Documentation

https://www.rabbitmq.com/amqp-0-9-1-reference.html#connection.forced

Workarounds

  1. 83% success Check broker resource alarms via `rabbitmq-diagnostics check_alarms` and free up memory/disk. For memory: reduce queue sizes or set `vm_memory_high_watermark` lower. For disk: increase `disk_free_limit`. 
    Check broker resource alarms via `rabbitmq-diagnostics check_alarms` and free up memory/disk. For memory: reduce queue sizes or set `vm_memory_high_watermark` lower. For disk: increase `disk_free_limit`.
  2. 78% success Implement connection recovery with backoff and listen to `connection.blocked` event. In Pika (Python): `connection.add_on_connection_blocked_callback(lambda c: time.sleep(10))`. 
    Implement connection recovery with backoff and listen to `connection.blocked` event. In Pika (Python): `connection.add_on_connection_blocked_callback(lambda c: time.sleep(10))`.

中文步骤

  1. Check broker resource alarms via `rabbitmq-diagnostics check_alarms` and free up memory/disk. For memory: reduce queue sizes or set `vm_memory_high_watermark` lower. For disk: increase `disk_free_limit`.
  2. Implement connection recovery with backoff and listen to `connection.blocked` event. In Pika (Python): `connection.add_on_connection_blocked_callback(lambda c: time.sleep(10))`.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Restarting the client without checking broker resources leads to immediate reconnection and re-disconnection.

  2. 90% fail

    Increasing connection timeout doesn't address the underlying resource exhaustion.

  3. 70% fail

    Adding more consumer instances worsens resource pressure by creating more connections.