kubernetes runtime_error ai_generated true

Error: Deployment rollout status is stuck: waiting for deployment spec update to be observed...

ID: kubernetes/deployment-rollout-stuck-image-pull-back-off

Also available as: JSON · Markdown · 中文
75%Fix Rate
82%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kubectl v1.28 active — — —
kubectl v1.29 active — — —
kubectl v1.30 active — — —

Root Cause

Deployment's image pull back-off prevents new pods from starting, so rollout never completes.

generic

中文

部署的镜像拉取回退阻止新 Pod 启动，因此回滚无法完成。

Official Documentation

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment

Workarounds

  1. 85% success Check image pull error: `kubectl describe pod <pod-name> | grep -A5 'Back-off'` to see exact reason (e.g., 'ImagePullBackOff'). Then fix image tag or registry credentials. 
    Check image pull error: `kubectl describe pod <pod-name> | grep -A5 'Back-off'` to see exact reason (e.g., 'ImagePullBackOff'). Then fix image tag or registry credentials.
  2. 90% success Update deployment with correct image: `kubectl set image deployment/<name> <container>=<correct-image>:<tag>` and watch rollout. 
    Update deployment with correct image: `kubectl set image deployment/<name> <container>=<correct-image>:<tag>` and watch rollout.
  3. 80% success If private registry, create image pull secret: `kubectl create secret docker-registry regcred --docker-server=<registry> --docker-username=<user> --docker-password=<pass> --docker-email=<email>` then patch service account. 
    If private registry, create image pull secret: `kubectl create secret docker-registry regcred --docker-server=<registry> --docker-username=<user> --docker-password=<pass> --docker-email=<email>` then patch service account.

中文步骤

  1. Check image pull error: `kubectl describe pod <pod-name> | grep -A5 'Back-off'` to see exact reason (e.g., 'ImagePullBackOff'). Then fix image tag or registry credentials.
  2. Update deployment with correct image: `kubectl set image deployment/<name> <container>=<correct-image>:<tag>` and watch rollout.
  3. If private registry, create image pull secret: `kubectl create secret docker-registry regcred --docker-server=<registry> --docker-username=<user> --docker-password=<pass> --docker-email=<email>` then patch service account.

Dead Ends

Common approaches that don't work:

  1. Force rollout restart with `kubectl rollout restart deployment` 90% fail

    Restart does not fix the underlying image pull issue; new pods will still fail.

  2. Delete and recreate the deployment 85% fail

    Recreating deployment uses same image spec; problem persists unless image is fixed.

  3. Wait longer (timeout >10 minutes) 80% fail

    Back-off is exponential; waiting alone won't resolve if image doesn't exist or auth fails.