kubernetes runtime_error ai_generated true

error: unable to upgrade connection: pod does not exist

ID: kubernetes/exec-in-terminating-pod

Also available as: JSON · Markdown · 中文
88%Fix Rate
85%Confidence
1Evidence
2023-05-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
1.24 active — — —
1.25 active — — —
1.26 active — — —
1.27 active — — —
1.28 active — — —

Root Cause

Attempting to exec into a pod that is in Terminating state or has already been deleted, causing the kubelet to reject the connection upgrade.

generic

中文

尝试对处于 Terminating 状态或已被删除的 Pod 执行 exec 操作，导致 kubelet 拒绝连接升级。

Official Documentation

https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/

Workarounds

  1. 85% success Wait for pod to fully terminate, then re-create: kubectl wait --for=delete pod/<pod-name> --timeout=60s && kubectl apply -f pod.yaml 
    Wait for pod to fully terminate, then re-create: kubectl wait --for=delete pod/<pod-name> --timeout=60s && kubectl apply -f pod.yaml
  2. 90% success Force delete the pod if stuck: kubectl delete pod <pod-name> --grace-period=0 --force, then exec into a new pod 
    Force delete the pod if stuck: kubectl delete pod <pod-name> --grace-period=0 --force, then exec into a new pod

中文步骤

  1. Wait for pod to fully terminate, then re-create: kubectl wait --for=delete pod/<pod-name> --timeout=60s && kubectl apply -f pod.yaml
  2. Force delete the pod if stuck: kubectl delete pod <pod-name> --grace-period=0 --force, then exec into a new pod

Dead Ends

Common approaches that don't work:

  1. Re-running kubectl exec immediately with same pod name 95% fail

    Pod 仍在终止中或已完全删除，重试不会改变状态

  2. Checking pod logs instead of exec 80% fail

    日志可能已被清理或 Pod 已不存在，日志命令同样会失败

  3. Restarting kubelet on the node 90% fail

    Pod 状态由 API Server 管理，kubelet 重启不影响已删除的 Pod