ImagePullBackOff cloud config_error ai_generated true

Pod stuck in ContainerCreating status: failed to pull image from ACR

ID: cloud/azure-aks-pod-stuck-containercreating

Also available as: JSON · Markdown · 中文
83%Fix Rate
86%Confidence
1Evidence
2024-05-18First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
AKS 1.28 active — — —
Azure CLI 2.58.0 active — — —
ACR Premium active — — —

Root Cause

Azure Kubernetes Service (AKS) pod cannot pull a container image from Azure Container Registry (ACR) because the AKS cluster's service principal or managed identity lacks AcrPull role assignment on the ACR, or the image tag does not exist.

generic

中文

Azure Kubernetes Service (AKS) Pod 无法从 Azure 容器注册表 (ACR) 拉取容器镜像，因为 AKS 集群的服务主体或托管标识缺少对 ACR 的 AcrPull 角色分配，或镜像标签不存在。

Official Documentation

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-image-pull-issues

Workarounds

  1. 90% success Assign the AcrPull role to the AKS cluster's managed identity: `az role assignment create --assignee <aks-cluster-identity-client-id> --role AcrPull --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ContainerRegistry/registries/<acr-name>`. Get the AKS identity via `az aks show -g <rg> -n <cluster-name> --query identityProfile.kubeletidentity.clientId`. 
    Assign the AcrPull role to the AKS cluster's managed identity: `az role assignment create --assignee <aks-cluster-identity-client-id> --role AcrPull --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ContainerRegistry/registries/<acr-name>`. Get the AKS identity via `az aks show -g <rg> -n <cluster-name> --query identityProfile.kubeletidentity.clientId`.
  2. 80% success If using a service principal, create a Kubernetes secret with ACR credentials and reference it in the pod spec: `kubectl create secret docker-registry acr-secret --docker-server=<acr-name>.azurecr.io --docker-username=<app-id> --docker-password=<password>`. Then add `imagePullSecrets: - name: acr-secret` to the deployment YAML. 
    If using a service principal, create a Kubernetes secret with ACR credentials and reference it in the pod spec: `kubectl create secret docker-registry acr-secret --docker-server=<acr-name>.azurecr.io --docker-username=<app-id> --docker-password=<password>`. Then add `imagePullSecrets: - name: acr-secret` to the deployment YAML.

中文步骤

  1. Assign the AcrPull role to the AKS cluster's managed identity: `az role assignment create --assignee <aks-cluster-identity-client-id> --role AcrPull --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ContainerRegistry/registries/<acr-name>`. Get the AKS identity via `az aks show -g <rg> -n <cluster-name> --query identityProfile.kubeletidentity.clientId`.
  2. If using a service principal, create a Kubernetes secret with ACR credentials and reference it in the pod spec: `kubectl create secret docker-registry acr-secret --docker-server=<acr-name>.azurecr.io --docker-username=<app-id> --docker-password=<password>`. Then add `imagePullSecrets: - name: acr-secret` to the deployment YAML.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    The same image pull configuration is used; the new pod will fail with the same error.

  2. 85% fail

    The kubelet uses the cluster's credential provider (e.g., acr-credential-provider), not the user's Docker config.