kubernetes runtime_error ai_generated true

AttachVolume.Attach failed for volume "pvc-xxx" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

ID: kubernetes/volume-attach-timeout-csi

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.24 active — — —
Kubernetes 1.27 active — — —
AWS EBS CSI Driver 1.12 active — — —
GCE PD CSI Driver 1.8 active — — —

Root Cause

CSI driver timed out while attaching a persistent volume to a node, often due to cloud provider API throttling or network issues between the CSI controller and storage backend.

generic

中文

CSI 驱动程序在将持久卷挂载到节点时超时，通常是由于云提供商 API 限流或 CSI 控制器与存储后端之间的网络问题。

Official Documentation

https://kubernetes.io/docs/concepts/storage/volumes/#csi

Workarounds

  1. 85% success Check CSI driver logs for specific errors: `kubectl logs -n kube-system <csi-controller-pod> csi-provisioner` and look for throttling or backend errors. 
    Check CSI driver logs for specific errors: `kubectl logs -n kube-system <csi-controller-pod> csi-provisioner` and look for throttling or backend errors.
  2. 75% success Increase the CSI driver's attach timeout by setting environment variable `ATTACH_TIMEOUT=5m` in the CSI controller deployment, then restart the pods. 
    Increase the CSI driver's attach timeout by setting environment variable `ATTACH_TIMEOUT=5m` in the CSI controller deployment, then restart the pods.
  3. 80% success Verify cloud provider API quotas (e.g., AWS EC2 API call rate limits) and request a quota increase if needed: `aws ec2 describe-account-attributes`. 
    Verify cloud provider API quotas (e.g., AWS EC2 API call rate limits) and request a quota increase if needed: `aws ec2 describe-account-attributes`.

中文步骤

  1. Check CSI driver logs for specific errors: `kubectl logs -n kube-system <csi-controller-pod> csi-provisioner` and look for throttling or backend errors.
  2. Increase the CSI driver's attach timeout by setting environment variable `ATTACH_TIMEOUT=5m` in the CSI controller deployment, then restart the pods.
  3. Verify cloud provider API quotas (e.g., AWS EC2 API call rate limits) and request a quota increase if needed: `aws ec2 describe-account-attributes`.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Restarting the CSI controller pod without checking cloud provider quotas or API rate limits will not resolve the underlying throttling issue.

  2. 50% fail

    Increasing CSI driver timeout without addressing storage backend latency only delays the failure.

  3. 60% fail

    Deleting and recreating the PVC/PV pair may be blocked by finalizers and does not fix the attach timeout root cause.