kubernetes network_error ai_generated true

http: TLS handshake error from 10.0.0.1:54321: remote error: tls: bad certificate

ID: kubernetes/kubelet-http-healthz-failure

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kubelet v1.28 active — — —
kubelet v1.29 active — — —
kubelet v1.30 active — — —

Root Cause

Kubelet's TLS certificate for healthz endpoint is expired or mismatched, causing API server to reject connection.

generic

中文

kubelet 用于 healthz 端点的 TLS 证书过期或不匹配，导致 API 服务器拒绝连接。

Official Documentation

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/

Workarounds

  1. 85% success Manually renew kubelet server certificate by running: `kubeadm init phase certs kubelet-server` and then restart kubelet. 
    Manually renew kubelet server certificate by running: `kubeadm init phase certs kubelet-server` and then restart kubelet.
  2. 90% success Check certificate expiry: `openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text -noout | grep -A2 Validity`. If expired, delete old cert and let kubelet auto-renew via CSR. 
    Check certificate expiry: `openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text -noout | grep -A2 Validity`. If expired, delete old cert and let kubelet auto-renew via CSR.
  3. 80% success Update kubelet configuration to use a valid certificate from a trusted CA by setting `--tls-cert-file` and `--tls-private-key-file` in /var/lib/kubelet/config.yaml. 
    Update kubelet configuration to use a valid certificate from a trusted CA by setting `--tls-cert-file` and `--tls-private-key-file` in /var/lib/kubelet/config.yaml.

中文步骤

  1. Manually renew kubelet server certificate by running: `kubeadm init phase certs kubelet-server` and then restart kubelet.
  2. Check certificate expiry: `openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text -noout | grep -A2 Validity`. If expired, delete old cert and let kubelet auto-renew via CSR.
  3. Update kubelet configuration to use a valid certificate from a trusted CA by setting `--tls-cert-file` and `--tls-private-key-file` in /var/lib/kubelet/config.yaml.

Dead Ends

Common approaches that don't work:

  1. Restart kubelet service without checking certificate renewal 90% fail

    Restarting does not regenerate expired certificates; they must be explicitly renewed or rotated.

  2. Disable TLS verification in kubelet config (--tls-cert-file=none) 95% fail

    Disabling TLS breaks secure communication and is not supported in production.

  3. Reinstall kubelet binary from scratch 85% fail

    Reinstallation does not fix certificate expiration; the certs are stored separately.