503 kubernetes runtime_error ai_generated true

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get apiservices.apiregistration.k8s.io v1.metrics.k8s.io)

ID: kubernetes/apiservice-not-available

Also available as: JSON · Markdown · 中文
78%Fix Rate
82%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.26 active — — —
Kubernetes 1.28 active — — —
metrics-server 0.6.3 active — — —

Root Cause

The API service for metrics (or another extension) is not responding, often because the corresponding controller (e.g., metrics-server) is down, misconfigured, or the service endpoint is unreachable.

generic

中文

指标（或其他扩展）的 API 服务未响应，通常是因为相应的控制器（如 metrics-server）已关闭、配置错误或服务端点不可达。

Official Documentation

https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server

Workarounds

  1. 85% success Check the status of the extension API server pod, e.g., for metrics-server: `kubectl get pods -n kube-system | grep metrics-server`. If down, check logs: `kubectl logs -n kube-system <metrics-server-pod>`. 
    Check the status of the extension API server pod, e.g., for metrics-server: `kubectl get pods -n kube-system | grep metrics-server`. If down, check logs: `kubectl logs -n kube-system <metrics-server-pod>`.
  2. 80% success Verify the service endpoint for the API service: `kubectl get endpoints -n kube-system <service-name>`. If no endpoints, the pod may not be listening on the correct port. 
    Verify the service endpoint for the API service: `kubectl get endpoints -n kube-system <service-name>`. If no endpoints, the pod may not be listening on the correct port.
  3. 75% success Delete and recreate the APIService resource to force re-registration: `kubectl delete apiservice v1.metrics.k8s.io && kubectl apply -f metrics-server-apiservice.yaml`. 
    Delete and recreate the APIService resource to force re-registration: `kubectl delete apiservice v1.metrics.k8s.io && kubectl apply -f metrics-server-apiservice.yaml`.

中文步骤

  1. Check the status of the extension API server pod, e.g., for metrics-server: `kubectl get pods -n kube-system | grep metrics-server`. If down, check logs: `kubectl logs -n kube-system <metrics-server-pod>`.
  2. Verify the service endpoint for the API service: `kubectl get endpoints -n kube-system <service-name>`. If no endpoints, the pod may not be listening on the correct port.
  3. Delete and recreate the APIService resource to force re-registration: `kubectl delete apiservice v1.metrics.k8s.io && kubectl apply -f metrics-server-apiservice.yaml`.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Restarting the kube-apiserver pod does not fix the issue because the problem is with the extension API server, not the core API server.

  2. 60% fail

    Reapplying the APIService YAML without checking the underlying service status may lead to the same error if the service is still down.