cicd build_optimization ai_generated true

Docker build cache miss: all layers rebuilt despite no source changes

ID: cicd/docker-layer-cache-invalidation

Also available as: JSON · Markdown
85%Fix Rate
85%Confidence
90Evidence
2020-06-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Docker layer cache is invalidated every CI run because COPY . . is placed before dependency installation, build arguments change every run, or the CI cache action isn't properly configured for BuildKit cache mounts.

generic

Workarounds

  1. 90% success Order Dockerfile to COPY dependency files first, install, then COPY source code 
    # Good Dockerfile ordering:
FROM node:20-slim
WORKDIR /app
COPY package.json package-lock.json ./  # Changes rarely
RUN npm ci                               # Cached unless deps change
COPY . .                                 # Source changes don't invalidate npm ci
RUN npm run build

    Sources: https://docs.docker.com/build/cache/

  2. 88% success Use docker/build-push-action with GitHub Actions cache backend 
    - uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

    Sources: https://docs.docker.com/build/cache/backends/gha/

  3. 85% success Use BuildKit cache mounts for package manager caches 
    # syntax=docker/dockerfile:1
FROM python:3.12-slim
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt
# The pip cache persists across builds, speeding up even when requirements change

    Sources: https://docs.docker.com/build/cache/#use-cache-mounts

Dead Ends

Common approaches that don't work:

  1. Adding --cache-from with the latest image tag without BuildKit 65% fail

    Legacy Docker builder's --cache-from only works if the image layers exactly match. Any change in a parent layer invalidates all subsequent layers. Without BuildKit's cache mount feature, dependency installation layers are rebuilt whenever any source file changes.

  2. Caching the entire /var/lib/docker directory in CI 70% fail

    Docker's storage driver (overlay2) creates many small files that are slow to save/restore from CI cache. The cache upload time often exceeds the time saved. GitHub Actions has a 10GB cache limit that fills quickly.

Error Chain

Preceded by:
Frequently confused with: