UPSTREAM_TIMEOUT cloud network_error ai_generated true

UPSTREAM_TIMEOUT: The request timed out when connecting to the VPC connector. Check VPC connector health and subnet capacity.

ID: cloud/gcp-cloud-run-request-timeout-vpc-connector

Also available as: JSON · Markdown · 中文
80%Fix Rate
87%Confidence
1Evidence
2023-09-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gcloud_cli active — — —
cloud_run active — — —
vpc_connector active — — —

Root Cause

The Cloud Run service's VPC connector (Serverless VPC Access) has exhausted its subnet IP addresses or is underprovisioned, causing connection timeouts.

generic

中文

Cloud Run 服务的 VPC 连接器（Serverless VPC Access）的子网 IP 地址已耗尽或配置不足，导致连接超时。

Official Documentation

https://cloud.google.com/vpc/docs/configure-serverless-vpc-access

Workarounds

  1. 85% success Increase the VPC connector's subnet size: `gcloud compute networks vpc-access connectors update my-connector --region us-central1 --min-instances 2 --max-instances 10` (if using scalable connector) or create a new connector with a /28 subnet (16 IPs) instead of /29 (8 IPs). 
    Increase the VPC connector's subnet size: `gcloud compute networks vpc-access connectors update my-connector --region us-central1 --min-instances 2 --max-instances 10` (if using scalable connector) or create a new connector with a /28 subnet (16 IPs) instead of /29 (8 IPs).
  2. 75% success Monitor VPC connector usage: `gcloud compute networks vpc-access connectors describe my-connector --region us-central1` and check `maxInstances` and `connectedProjects`. If near limits, deploy a second connector in a different subnet and route traffic via DNS. 
    Monitor VPC connector usage: `gcloud compute networks vpc-access connectors describe my-connector --region us-central1` and check `maxInstances` and `connectedProjects`. If near limits, deploy a second connector in a different subnet and route traffic via DNS.

中文步骤

  1. Increase the VPC connector's subnet size: `gcloud compute networks vpc-access connectors update my-connector --region us-central1 --min-instances 2 --max-instances 10` (if using scalable connector) or create a new connector with a /28 subnet (16 IPs) instead of /29 (8 IPs).
  2. Monitor VPC connector usage: `gcloud compute networks vpc-access connectors describe my-connector --region us-central1` and check `maxInstances` and `connectedProjects`. If near limits, deploy a second connector in a different subnet and route traffic via DNS.

Dead Ends

Common approaches that don't work:

  1. 85% fail

    The timeout is at the VPC connector level, not the application; increasing the Cloud Run timeout doesn't affect the connector's IP exhaustion.

  2. 90% fail

    More instances increase demand on the VPC connector, worsening IP exhaustion.