The Reproducibility Problem with Dockerfiles

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/chainguard-image-toolchain/viewer/05-problem.

Let’s first confirm the problems melange and apko are trying to solve. The Dockerfiles most teams write day to day have structural weaknesses in three areas: reproducibility, attack surface, and SBOM accuracy.

Depending on the state of a repository at build time

A typical Dockerfile installs packages with a RUN instruction like this:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl ca-certificates

If you rebuild this image from the same Dockerfile a year later, the state of the package repository that apt-get update refers to has changed since the last build. Even if you pin the base image tag, package versions still shift to whatever the latest version happens to be on the repository’s mirrors. As a result, the same Dockerfile can’t reliably produce a bit-for-bit identical image every time. Pinning package versions mitigates this, but pinning every dependency of every dependency isn’t realistic, so most Dockerfiles leave this problem unaddressed.

Freedom of instructions widens the attack surface

RUN can execute any shell command. That freedom is convenient for developers, but it makes it hard to mechanically account for what actually happened inside the image. Dockerfiles that fetch a script from the internet and run it directly, in a pattern like curl | sh install.sh, aren’t unusual — but at that point, the build process stops being verifiable.

On top of that, basing an image on a general-purpose distribution like Ubuntu or Debian bundles in a shell, a package manager, and various utilities the application never needs at runtime. Even if these aren’t used at runtime, a scanning tool will flag them the moment a vulnerability is found in them, making them targets for patching. For an attacker, a container with a working shell is also an easier foothold after a break-in.

Reconstructing an accurate SBOM after the fact is hard

An SBOM (Software Bill of Materials) is a list of the packages an image contains and their versions. In recent years, more organizations have started requiring SBOMs to ensure transparency in the software supply chain.

To produce an SBOM for an image built from a Dockerfile, the common approach is to scan the image’s filesystem after the build finishes, and infer package information from package manager databases (such as dpkg or apk database files) or from version strings embedded in binaries. Syft’s binary cataloger, for example, matches binary filename patterns and known string patterns embedded in file contents (such as Ruby’s embedded string ruby 3.4.0dev (...)) using regular expressions, and infers a package name and version from the match. Because this method is an after-the-fact inference done from outside the build process, it can’t reliably capture software compiled from source inside a RUN instruction, or binaries placed without going through a package manager.

The melange and apko approach

melange and apko answer these three problems by redesigning the build process itself. melange describes each build step as a declarative pipeline, and generates a fragment of the SBOM at package build time, embedding it inside the package. apko doesn’t allow running arbitrary commands the way RUN does; it composes an image purely from a combination of APK packages, making the build process itself deterministic. The next chapter starts by looking at Wolfi, the distribution these two tools use as their primary material.