kubernetes network_error ai_generated true

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "pod-identity-webhook.m8r.io": Post "https://pod-identity-webhook.m8r.io/mutate?timeout=10s": context deadline exceeded

ID: kubernetes/mutating-webhook-timeout

Also available as: JSON · Markdown · 中文
85%Fix Rate
87%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.26 active — — —
Kubernetes 1.27 active — — —
Kubernetes 1.28 active — — —
istio 1.18 active — — —
cert-manager 1.12 active — — —

Root Cause

A mutating admission webhook is unreachable or timing out, often due to network issues, missing service, or the webhook pod being down.

generic

中文

可变准入 Webhook 不可达或超时，通常是由于网络问题、服务缺失或 Webhook Pod 宕机。

Official Documentation

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

Workarounds

  1. 85% success Check webhook pod status: kubectl get pods -n <namespace> -l app=<webhook-name>, then restart if not running: kubectl rollout restart deployment/<webhook-deployment> -n <namespace> 
    Check webhook pod status: kubectl get pods -n <namespace> -l app=<webhook-name>, then restart if not running: kubectl rollout restart deployment/<webhook-deployment> -n <namespace>
  2. 80% success Verify network connectivity by exec into a temporary pod: kubectl run test --image=busybox --rm -it -- wget -O- https://pod-identity-webhook.m8r.io/mutate --timeout=5 
    Verify network connectivity by exec into a temporary pod: kubectl run test --image=busybox --rm -it -- wget -O- https://pod-identity-webhook.m8r.io/mutate --timeout=5

中文步骤

  1. Check webhook pod status: kubectl get pods -n <namespace> -l app=<webhook-name>, then restart if not running: kubectl rollout restart deployment/<webhook-deployment> -n <namespace>
  2. Verify network connectivity by exec into a temporary pod: kubectl run test --image=busybox --rm -it -- wget -O- https://pod-identity-webhook.m8r.io/mutate --timeout=5

Dead Ends

Common approaches that don't work:

  1. Increasing webhook timeout in the ValidatingWebhookConfiguration 75% fail

    超时通常是网络不通或服务宕机，增加超时不会解决根本问题

  2. Restarting the API server 90% fail

    问题出在 Webhook 服务端，不是 API 服务器

  3. Disabling the webhook temporarily by deleting the MutatingWebhookConfiguration 50% fail

    虽然可以绕过错误，但会禁用安全功能，且需要重新创建配置