137 cloud resource_error ai_generated partial

Container terminated: Exit code 137 (Out of Memory)

ID: cloud/gcp-cloud-run-container-crash-oom

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2023-09-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Cloud Run (fully managed) active — — —
Cloud Run for Anthos active — — —
gcloud CLI 460.0.0 active — — —

Root Cause

Cloud Run container exceeds its allocated memory limit (default 512 MB), causing the kernel OOM killer to terminate the process.

generic

中文

Cloud Run 容器超过其分配的内存限制（默认 512 MB），导致内核 OOM 杀手终止进程。

Official Documentation

https://cloud.google.com/run/docs/troubleshooting#container-crash-oom

Workarounds

  1. 85% success Increase the memory limit for the Cloud Run service: `gcloud run deploy <SERVICE> --memory=1Gi --region=us-central1`. Monitor memory usage via Cloud Monitoring and adjust accordingly. Also consider reducing concurrency: `gcloud run deploy <SERVICE> --concurrency=10` to limit simultaneous requests. 
    Increase the memory limit for the Cloud Run service: `gcloud run deploy <SERVICE> --memory=1Gi --region=us-central1`. Monitor memory usage via Cloud Monitoring and adjust accordingly. Also consider reducing concurrency: `gcloud run deploy <SERVICE> --concurrency=10` to limit simultaneous requests.
  2. 75% success Profile the application to find memory-heavy operations. For Node.js, use `--max-old-space-size` flag in the Dockerfile: `CMD ["node", "--max-old-space-size=256", "app.js"]`. For Python, use `gc.set_threshold()` to tune garbage collection. 
    Profile the application to find memory-heavy operations. For Node.js, use `--max-old-space-size` flag in the Dockerfile: `CMD ["node", "--max-old-space-size=256", "app.js"]`. For Python, use `gc.set_threshold()` to tune garbage collection.

中文步骤

  1. Increase the memory limit for the Cloud Run service: `gcloud run deploy <SERVICE> --memory=1Gi --region=us-central1`. Monitor memory usage via Cloud Monitoring and adjust accordingly. Also consider reducing concurrency: `gcloud run deploy <SERVICE> --concurrency=10` to limit simultaneous requests.
  2. Profile the application to find memory-heavy operations. For Node.js, use `--max-old-space-size` flag in the Dockerfile: `CMD ["node", "--max-old-space-size=256", "app.js"]`. For Python, use `gc.set_threshold()` to tune garbage collection.

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Memory leaks continue to grow unbounded; the container will eventually hit the new limit too, wasting money.

  2. 95% fail

    The same container image with the same memory limit will crash again under the same load.