K8S-HTTP-503 kubernetes runtime_error ai_generated true

http: HTTP probe failed with statuscode: 503

ID: kubernetes/kubelet-http-healthz-fail

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kubernetes 1.22 active
kubernetes 1.23 active
kubernetes 1.27 active
kubernetes 1.28 active

Root Cause

The application's HTTP health endpoint returned a 503 status, usually because the app is not ready to serve traffic or is overloaded.

generic

中文

应用程序的 HTTP 健康检查端点返回 503 状态,通常是因为应用未准备好提供服务或过载。

Official Documentation

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Workarounds

  1. 90% success Check the application logs for errors during startup or dependency failures. For example, run `kubectl logs <pod-name> -c <container>` and look for stack traces.
    Check the application logs for errors during startup or dependency failures. For example, run `kubectl logs <pod-name> -c <container>` and look for stack traces.
  2. 80% success Ensure the health endpoint returns 200 only when all dependencies (database, cache) are ready. Use a startup probe with a longer initial delay if the app takes time to initialize.
    Ensure the health endpoint returns 200 only when all dependencies (database, cache) are ready. Use a startup probe with a longer initial delay if the app takes time to initialize.

中文步骤

  1. 检查应用程序日志中的启动错误或依赖项故障。例如,运行 `kubectl logs <pod-name> -c <container>` 并查找堆栈跟踪。
  2. 确保健康检查端点仅在所有依赖项(数据库、缓存)就绪时返回 200。如果应用程序需要时间初始化,请使用具有较长初始延迟的启动探测。

Dead Ends

Common approaches that don't work:

  1. Increase the liveness probe threshold without fixing the app. 70% fail

    This only masks the problem; the app remains unhealthy and may crash later.

  2. Remove the readiness probe entirely. 85% fail

    Removing the readiness probe can cause traffic to be routed to unhealthy pods, leading to user-facing errors.