HTTP 503 cloud runtime_error ai_generated partial

HTTP 503 Service Unavailable: The request failed because the service is scaling up. Try again later.

ID: cloud/gcp-cloud-run-cold-start-http-503

Also available as: JSON · Markdown · 中文
80%Fix Rate
87%Confidence
1Evidence
2023-09-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Cloud Run (fully managed) gen2 active — — —
Cloud Run for Anthos 1.28 active — — —

Root Cause

Cloud Run's cold start latency (due to container image pull and startup) exceeds the request timeout, causing the load balancer to return 503 before the container is ready.

generic

中文

Cloud Run 的冷启动延迟（因容器镜像拉取和启动）超过请求超时时间，导致负载均衡器在容器就绪前返回 503。

Official Documentation

https://cloud.google.com/run/docs/troubleshooting#503-errors

Workarounds

  1. 85% success Set min instances to at least 1 to keep a warm instance always ready: gcloud run deploy SERVICE --min-instances 1. For production, use 2-3 to handle traffic spikes. 
    Set min instances to at least 1 to keep a warm instance always ready: gcloud run deploy SERVICE --min-instances 1. For production, use 2-3 to handle traffic spikes.
  2. 80% success Optimize container startup: use distroless base images, reduce image size, and move initialization to a background thread. Example Dockerfile: FROM gcr.io/distroless/java17-debian11, then use Spring Boot's lazy initialization. 
    Optimize container startup: use distroless base images, reduce image size, and move initialization to a background thread. Example Dockerfile: FROM gcr.io/distroless/java17-debian11, then use Spring Boot's lazy initialization.
  3. 75% success Enable 'startup CPU boost' to allocate additional CPU during container startup: gcloud run deploy SERVICE --cpu-boost 
    Enable 'startup CPU boost' to allocate additional CPU during container startup: gcloud run deploy SERVICE --cpu-boost

中文步骤

  1. 将最小实例数设置为至少 1 以保持一个常驻实例：gcloud run deploy SERVICE --min-instances 1。生产环境建议设为 2-3 以应对流量峰值。
  2. 优化容器启动：使用 distroless 基础镜像、减小镜像体积、将初始化移至后台线程。示例 Dockerfile：FROM gcr.io/distroless/java17-debian11，然后使用 Spring Boot 的懒加载。
  3. 启用 'startup CPU boost' 在容器启动期间分配额外 CPU：gcloud run deploy SERVICE --cpu-boost

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Simply retrying the request without addressing cold start may succeed eventually but adds latency and costs.

  2. 95% fail

    Increasing max instances doesn't reduce cold start frequency; it only limits concurrency.

  3. 60% fail

    Setting min instances to 1 reduces cold start for the first instance but doesn't help if all instances are busy.