K8S_CONFLICT cicd config_error ai_generated true

Error from server (Conflict): Operation cannot be fulfilled on pods "my-app-5d4f8b7c6-abcde": the object has been modified; please apply your changes to the latest version and try again

ID: cicd/kubectl-apply-pending

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2024-03-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.28 active — — —
Kubernetes 1.29 active — — —
kubectl 1.29.0 active — — —

Root Cause

A Kubernetes resource update conflict occurs when two or more controllers or CI jobs try to modify the same resource concurrently, causing the second update to fail due to stale resourceVersion.

generic

中文

当两个或多个控制器或 CI 作业尝试同时修改同一 Kubernetes 资源时，会发生资源更新冲突，导致第二个更新因 resourceVersion 过期而失败。

Official Documentation

https://kubernetes.io/docs/reference/using-api/api-concepts/#conflicts

Workarounds

  1. 80% success Re-run the CI job after a short delay (e.g., 10 seconds) to allow the conflicting operation to complete and the resourceVersion to update. 
    Re-run the CI job after a short delay (e.g., 10 seconds) to allow the conflicting operation to complete and the resourceVersion to update.
  2. 90% success Use `kubectl apply --server-side=true` which uses server-side apply to merge changes without conflict errors. 
    Use `kubectl apply --server-side=true` which uses server-side apply to merge changes without conflict errors.
  3. 85% success Add a retry loop in the CI script: `for i in 1 2 3; do kubectl apply -f deployment.yaml && break || sleep 5; done` 
    Add a retry loop in the CI script: `for i in 1 2 3; do kubectl apply -f deployment.yaml && break || sleep 5; done`

中文步骤

  1. 在短暂延迟（例如 10 秒）后重新运行 CI 作业，以允许冲突操作完成并更新 resourceVersion。
  2. 使用 `kubectl apply --server-side=true`，它使用服务器端应用来合并更改而不会出现冲突错误。
  3. 在 CI 脚本中添加重试循环：`for i in 1 2 3; do kubectl apply -f deployment.yaml && break || sleep 5; done`

Dead Ends

Common approaches that don't work:

  1. 60% fail

    Deleting and recreating the resource manually; this causes downtime and may trigger cascading failures in dependent services.

  2. 70% fail

    Using `kubectl replace --force` which deletes and recreates the resource; this bypasses conflict detection but can delete other dependent objects.

  3. 90% fail

    Ignoring the error and retrying the same CI job without changes; the conflict persists because the resourceVersion hasn't been updated.