kubernetes system_error ai_generated true

Error: failed to start container: failed to create containerd task: mount /var/lib/kubelet/pods/.../volumes/...: operation not permitted

ID: kubernetes/volume-mount-readonly-filesystem

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.28 active — — —
Kubernetes 1.29 active — — —
containerd 1.7.0 active — — —
Ubuntu 22.04 active — — —

Root Cause

Container runtime (containerd) cannot mount a volume because the underlying filesystem is read-only or the mount point is invalid, often due to SELinux enforcement or AppArmor profile.

generic

中文

容器运行时（containerd）无法挂载卷，因为底层文件系统是只读的或挂载点无效，通常由于SELinux强制或AppArmor配置文件。

Official Documentation

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Workarounds

  1. 90% success Check SELinux status: `getenforce`. If enforcing, temporarily set to permissive: `setenforce 0`. Then verify mount succeeds. For persistent fix, adjust SELinux policy: `semanage fcontext -a -t container_file_t '/var/lib/kubelet/pods/.*/volumes/.*(/.*)?'`. 
    Check SELinux status: `getenforce`. If enforcing, temporarily set to permissive: `setenforce 0`. Then verify mount succeeds. For persistent fix, adjust SELinux policy: `semanage fcontext -a -t container_file_t '/var/lib/kubelet/pods/.*/volumes/.*(/.*)?'`.
  2. 80% success Check AppArmor status: `aa-status`. If a profile is blocking, unload it: `aa-remove-unknown` or create a custom profile allowing mount operations. 
    Check AppArmor status: `aa-status`. If a profile is blocking, unload it: `aa-remove-unknown` or create a custom profile allowing mount operations.

中文步骤

  1. 检查SELinux状态：`getenforce`。如果是强制模式，临时设置为宽松模式：`setenforce 0`。然后验证挂载是否成功。持久修复：调整SELinux策略：`semanage fcontext -a -t container_file_t '/var/lib/kubelet/pods/.*/volumes/.*(/.*)?'`。
  2. 检查AppArmor状态：`aa-status`。如果有配置文件阻止，卸载它：`aa-remove-unknown`或创建允许挂载操作的自定义配置文件。

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Restarting kubelet or containerd doesn't fix the underlying filesystem or SELinux issue; mounts will still fail.

  2. 80% fail

    Re-creating the pod without fixing SELinux context or AppArmor profile results in the same mount error.