php runtime_error ai_generated partial

Illuminate\Queue\MaxAttemptsExceededException: App\Jobs\SendNotification has been attempted too many times or run too long. The job may have been previously processed by a worker that was forcefully terminated.

ID: php/laravel-queue-worker-sigterm-timeout

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-07-22First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
laravel 10 active — — —
laravel 11 active — — —
php 8.2 active — — —
php 8.3 active — — —
horizon 5.21 active — — —

Root Cause

A Laravel queue worker received a SIGTERM signal (e.g., during deployment or scaling) while processing a job, causing the job to be marked as failed or re-attempted without proper release, exceeding the maximum attempts limit.

generic

中文

Laravel 队列工作进程在处理作业时收到 SIGTERM 信号（例如在部署或扩展期间），导致作业被标记为失败或重新尝试而未正确释放，超过了最大尝试次数限制。

Official Documentation

https://laravel.com/docs/11/queues#max-attempts

Workarounds

  1. 90% success Implement a `failed` method in the job to handle the exception gracefully, and use `php artisan queue:retry all` to re-run failed jobs after ensuring workers are stable. 
    Implement a `failed` method in the job to handle the exception gracefully, and use `php artisan queue:retry all` to re-run failed jobs after ensuring workers are stable.
  2. 85% success Add a `pcntl_signal` handler in the job's `handle` method to catch SIGTERM and release the job back to the queue: `pcntl_signal(SIGTERM, function () { $this->release(30); });` 
    Add a `pcntl_signal` handler in the job's `handle` method to catch SIGTERM and release the job back to the queue: `pcntl_signal(SIGTERM, function () { $this->release(30); });`

中文步骤

  1. Implement a `failed` method in the job to handle the exception gracefully, and use `php artisan queue:retry all` to re-run failed jobs after ensuring workers are stable.
  2. Add a `pcntl_signal` handler in the job's `handle` method to catch SIGTERM and release the job back to the queue: `pcntl_signal(SIGTERM, function () { $this->release(30); });`

Dead Ends

Common approaches that don't work:

  1. Increasing the `--tries` option on the queue worker to a very high number 80% fail

    The issue is not about the number of attempts but about the job being incorrectly marked as failed due to SIGTERM; increasing tries may delay the failure but not prevent it.

  2. Disabling the `force` option in deploy scripts that send SIGTERM 70% fail

    SIGTERM is necessary for graceful shutdown; disabling it can cause longer downtimes or zombie processes.