You edit a line in a Markdown file. You push to CI. Suddenly Docker downloads all your npm packages from scratch. The README change never ran inside a container. It never affected a RUN command. Yet every layer after the COPY step rebuilt. This is not a bug. It is the build cache working exactly as designed. The surprising part is not that it happens. It is that most explanations make it sound like it should not.
The Dockerfile defines a sequence of transformations. Each instruction takes a filesystem snapshot, applies a change, and produces a new snapshot. A layer. The builder caches these layers by computing a key from the exact inputs to each step. The key includes the parent layer’s digest. A hash of the layer’s contents. If any earlier layer changes even slightly, its digest changes. That new digest becomes part of the key for every instruction below it. The builder refuses to reuse a cached layer that was built on a different parent. This is the same principle as a blockchain where each block commits to the hash of the previous one. One altered block forces the entire chain to rebuild.
This safety model guarantees reproducibility. The builder cannot know if a changed earlier layer actually affects later steps. Network calls, environment variables, side effects. It would have to re-execute the whole pipeline to find out. So it assumes that every subsequent step could depend on the new state. The only correct cache hit is one where every input, including the parent filesystem, is identical byte for byte. That means a cache miss anywhere upstream invalidates every downstream layer in the same stage. Even layers whose Dockerfile lines you never touched.
The Cache Is a Content‑Addressable Hash Chain
Think of each layer as a snapshot identified by a cryptographic hash of its contents. The hash is immutable. Change one byte and you get a completely different hash. The image manifest lists these hashes in order. When you run a container, the filesystem driver stacks them. The topmost layer overrides files from below. Read operations search downward. Writes go to a thin writable container layer.
The same stacking happens during a build. Docker starts with the base image from FROM. That filesystem has a digest. The first instruction generates a new layer by applying its operation to that base. The builder computes a cache key that combines the base digest, the instruction text, and any external inputs like file checksums. If it has already built an identical layer from those exact inputs, it skips the work. It reuses the layer and its digest. The next instruction then uses that digest as its parent, and the cycle continues.
Because the parent digest is part of the key, the chain is not a set of independent caches. It is a linked list of dependencies. If the layer at position 5 changes, its digest now differs from the one used to build layer 6 in the previous run. The cached layer 6 was created on top of the old digest. The new build offers a new parent digest, so the builder cannot match the old key. It must re-execute instruction 6. That produces a new layer with a new digest, breaking the match for instruction 7. This cascade continues until the stage ends or another stage starts fresh.
The builder does not attempt to inspect whether instruction 6 could actually produce the same output with the new parent. That would require running it first or building a sophisticated dependency graph. It would also break the content‑addressable purity model. The resulting image may look identical, but its digest would be different. The cache exists to avoid repeating work, not to guess when work is unnecessary. The conservative approach is the only one that never produces a subtly incorrect image.
What Actually Goes into a Cache Key
Not all instructions are equal. Some keys include more volatile inputs than others. Understanding which ones change the parent digest is the key to predicting rebuilds.
FROM pins you to a base image digest. Tags like ubuntu:22.04 are mutable pointers. Docker resolves them to a digest at build time. If that digest changes, either because you pulled a newer image or used --pull, the FROM cache misses. Every instruction in that stage now gets a new parent digest from the start. A full rebuild. This is often desirable for security patches, but it can surprise teams that do not pin their base images.
RUN stores the exact shell command string in the key. A single added space, a reordered flag, or a different line break means a different key. Even if the command’s filesystem side effects are identical, the builder treats it as a separate operation. That is safe. Shell commands can have arbitrary side effects that depend on timing, environment, or network.
COPY and ADD are the most deceptive. Their keys include checksums of every file they touch from the build context. Not just content. Permissions, ownership, and directory structure matter. Timestamps like last‑modified are ignored, but everything else is hashed. If you write COPY . /app/, any file change, anywhere in the entire context tree that is not excluded by .dockerignore, changes the checksum set. That one README adjustment, a new generated log, or a changed file permission invalidates the copy layer. The new layer gets a new digest, and all downstream layers rebuild.
ARG and ENV inject values into the environment. The cache key for ARG uses the argument name and value at the line where the ARG is declared. Changing an ARG value, even one that you never reference directly in a later RUN, invalidates that ARG instruction and everything after it. ENV behaves similarly. If a later RUN uses an environment variable that changed, it will of course miss the cache. But the key is broader: the presence of a new ARG value alone is enough to create a new parent digest for subsequent instructions, because those instructions execute with that value in their environment. The builder cannot prove they do not use it.
Real‑World Cascade Scenarios
The most common surprise is the monolithic COPY . . early in a Dockerfile. A developer adds a new test fixture file. No code changes. The next docker build sees a different checksum set for the copy step. That step rebuilds. Because it comes before RUN npm install, the entire dependency download and installation re‑executes. The fix is to copy only what you need, in order of how often it changes. Package manifests first, then source code later. That way, dependency layers stay cached as long as the lock files do not change.
Another common trigger is a CI agent that does not share a persistent cache. Each job pulls a clean environment. The local Docker layer store is empty. If you do not use an external cache backend with --cache-from, every build starts without any previous layers. The builder cannot find a matching parent digest for anything. A full rebuild. The Dockerfile did not change. The source tree did not change. The cache was simply missing.
Build arguments injected from CI variables cause silent rebuilds. A build timestamp, a Git SHA, or a version number passed via --build-arg invalidates everything from that ARG declaration downward. If that ARG appears early, you pay for a full re‑execution. If you only need that value for labeling or a final RUN, move the ARG as late as possible in the Dockerfile. That limits the blast radius.
Base image updates that you do not control also cascade. If you build on a hosted CI that caches the base image locally but updates it periodically, your build may suddenly rebuild everything because the pinned tag now resolves to a new digest. Pinning to a digest with FROM ubuntu@sha256:... eliminates that volatility but means you have to manually bump for updates.
BuildKit Adds Graph‑Level Intelligence, Not Semantic Shortcuts
BuildKit’s solver represents the build as a dependency graph rather than a linear list of instructions. It can parallelize independent steps and prune unused branches in multi‑stage builds. It can also export cache metadata to registries or cloud storage and import it on other machines via --cache-from and --cache-to. This lets you share caches across ephemeral CI workers, which solves the empty‑cache problem.
But BuildKit does not break the sequential invalidation rule inside a single stage. A COPY layer that changes will still force a new parent digest for the subsequent RUN. The internal vertex for that RUN will look for a cache hit using the new parent digest. Since the previous cache entry was built with a different parent, it is not eligible for reuse. The graph structure may allow other unrelated stages to remain cached, but the linear chain within a stage remains intact.
The newer docker-container driver and remote cache backends simply preserve that chain across machines. They do not make the builder reason about which downstream steps are “truly” affected. That would require understanding the semantics of every shell command and programming language build system. The performance benefits of graph‑based scheduling and shared caches are significant. The invalidation logic, however, still follows the same content‑addressable purity.
Diagnosing Unexpected Rebuilds
The fastest way to confirm a cascade is to look at the build output. When the first step that rebuilds is not the one you changed, the cause is upstream. If step 3 was cached and step 4 rebuilt, inspect what step 4 depends on. Check if the base image digest changed. Compare the build context checksums if it is a COPY. Use docker build --no-cache only for debugging, not as a workaround.
For CI, enable BuildKit’s inline cache or registry cache and compare cache imports. The max cache mode exports all intermediate layers, making subsequent imports more likely to find matching digests. Without that, only the final image layers are pushed, which may not include the exact parent digest for a later step. That can cause false rebuilds because the required parent layer was simply not exported.
The .dockerignore file is your first defense. Exclude .git, node_modules, test reports, and any generated files that change frequently. The build context tarball that Docker ships to the daemon becomes smaller, and the checksum surface for COPY instructions shrinks. This reduces the odds that an unrelated file change invalidates a layer you intended to be stable.
Use multi‑stage builds to isolate the cascade. A front‑end build stage can produce static assets. The final stage copies only that output. If the toolchain changes, only the relevant stage rebuilds; the final image remains cached. This modularizes the hash chains and limits the distance a cache miss can travel.
Quick Reference: Cache Invalidation by Instruction
| Instruction | Cache Key Inputs | Common Miss Triggers |
|---|---|---|
FROM | Base image digest | Tag moved, --pull fetches new image |
RUN | Parent digest, exact command string | Any change to the line, including whitespace |
COPY/ADD | Parent digest, file content + permission checksums | Any modification to copied files or their metadata |
ARG | Parent digest, argument name and value at declaration | Value changes between builds, even if not referenced by RUN |
ENV | Parent digest, variable name and value | Value changes, variable added or removed |
WORKDIR, USER, etc. | Parent digest, parameters | Parameter changes or any parent digest change |
Frequently Asked Questions
Q: Why can’t Docker just see that my later RUN is independent and skip rebuilding it?
The builder lacks semantic knowledge of shell commands. It cannot predict whether a changed parent filesystem will affect the output of a RUN that does not read that changed file. The only safe guarantee is that identical inputs produce identical outputs. By re‑executing the step, it preserves that purity. Any shortcut would risk images that look correct but differ in subtle ways, breaking reproducibility.
Q: How do I stop a README change from rebuilding my npm install step?
Split your Dockerfile so that dependency installation happens before you copy the full source tree. Copy package.json and package-lock.json first, run npm install, then copy the rest of the source. The README lives in the second copy. The dependency layer remains cached as long as the lock files do not change.
Q: Does BuildKit eliminate the sequential invalidation problem?
No. Inside a single stage, the cache chain still depends on parent digests. BuildKit gives you graph‑level parallelism, remote caches, and more flexible export modes. It can avoid rebuilding unrelated stages, but within a stage, the dominoes still fall. A changed COPY still forces all later layers to rebuild unless they find a remote cache hit with that exact new parent.
Q: How do ARG values cause cache misses even when the RUN command doesn’t use them?
The ARG is declared and its value becomes part of the environment for all subsequent instructions. Docker treats the instruction as having a dependency on that value, because the environment is part of the input state. Changing the value changes the ARG layer’s digest, which then changes the parent digest for the next RUN. The builder conservatively assumes the RUN could be influenced, so the cache does not match.
Q: Why does a fresh CI agent build everything from scratch even with an identical Dockerfile and source?
The local layer cache is empty. Without importing an external cache via --cache-from, the builder has no previous results to match against. Even if the inputs are identical, the builder has no record of having built them before. You need to export the cache from a previous build and import it in the new environment. BuildKit’s registry or GitHub Actions cache backends make this straightforward.
Never Let the Cache Cascade Surprise You Again
The Docker build cache is a hash chain, not a smart dependency tracker. It trades sophistication for correctness. Once you internalize that model, the “random” rebuilds become predictable. You start seeing your Dockerfile as a sequence of commits whose hashes depend on all ancestors. You learn which instructions touch the most volatile inputs and how to reorder them to protect expensive steps.
Understanding the internals of the tools you use every day is the fastest way to stop fighting them. If you want breakdowns like this delivered straight to your inbox, subscribe to Internals Decoded at internalsdecoded.com. Every issue explains a real system, not a tutorial rehash, so you can stop guessing and start reasoning from first principles.