On reproducible Docker images

multi-arch Docker images without immutable tags

· Dmytro Shteflyuk

Most of the base images in the Docker registry do not offer immutable tags. This means that if you’re building your Docker image based on, for example, an official Ruby image ruby:3.3.0-slim, it might change overnight, and the machines that have already downloaded this tag will never receive an updated image.

Table of contents

Why repository maintainers do this?

The short answer is to pick up package security updates. Software is usually released much less frequently, and leaving containers as they are could lead to people using vulnerable base images. Sometimes developers can release a bug fix without bumping the version of the software; for example, see a ruby update that addresses a 3.3.0 crash on ARM64 architecture.

Here are some threads where people are asking what is going on:

Docker official library FAQ answers the question about what happens after the source code changes. In essence, the new image will be built, pushed to the repository, and re-tagged with the same tag.

Reproducible builds

This opens an interesting case when somebody already has pulled the previous image. If the base image was updated to address a security vulnerability or a critical bug, they will never know it until they explicitly pull the tag again.

Another problem might arise after the build succeeds on a local machine, but mysteriously fails on CI because the base image packages have different versions.

To solve this, the official guidance is to use manifest digest:

FROM registry.docker.com/library/ruby:3.3.0-slim@sha256:cdf1bae55aaa4ed3c9927ed6f67f6f35e9de4d6b6a2d29249411937b49426034 AS base

In this case, Docker will ignore the tag (we can still use it as a hint for the reader, just need to make sure we change the version when the manifest digest is updated). We can obtain the digest by browsing Docker Hub or by running manifest-tool:

manifest-tool inspect registry.docker.com/library/ruby:3.3.0-slim

The output will look like:

Name:   registry.docker.com/library/ruby:3.3.0-slim (Type: application/vnd.oci.image.index.v1+json)
Digest: sha256:abcb7c3943a085f511397b65ba7ca32ad56af759a53af932c5e354e1a8f84bcb
 * Contains 16 manifest references (8 images, 8 attestations):
[1]     Type: application/vnd.oci.image.manifest.v1+json
[1]   Digest: sha256:cdf1bae55aaa4ed3c9927ed6f67f6f35e9de4d6b6a2d29249411937b49426034
[1]   Length: 1934
[1] Platform:
[1]    -      OS: linux
[1]    -    Arch: amd64
[1] # Layers: 5
...
[7]     Type: application/vnd.oci.image.manifest.v1+json
[7]   Digest: sha256:ee13c5706b5e817521cbd1572d7aa35696d4b9c721340ba36827e58dcc22852e
[7]   Length: 1936
[7] Platform:
[7]    -      OS: linux
[7]    -    Arch: arm64
[7]    - Variant: v8
[7] # Layers: 5
...

In addition, there is an experimental feature in Docker that allows to manipulate manifests with docker CLI:

docker manifest inspect registry.docker.com/library/ruby:3.3.0-slim

At the moment, it will print in JSON the same information available on the website.

Multi-arch images

Now, we have a solution for reproducible builds for single-platform builds. But how will this work for multi-platform builds? For example, Kamal builds a multi-platform image with buildx:

docker buildx build --platform linux/arm64,linux/amd64 .

The answer lies in the first lines of the manifest-tool output:

Name:   registry.docker.com/library/ruby:3.3.0-slim (Type: application/vnd.oci.image.index.v1+json)
Digest: sha256:abcb7c3943a085f511397b65ba7ca32ad56af759a53af932c5e354e1a8f84bcb
 * Contains 16 manifest references (8 images, 8 attestations):

We can actually use the digest of the manifest itself to reference a multi-platform base image:

FROM registry.docker.com/library/ruby@sha256:abcb7c3943a085f511397b65ba7ca32ad56af759a53af932c5e354e1a8f84bcb AS base

Bonus: deploying Ruby 3.3.0 on ARM64

If you have tried to use Ruby 3.3.0 to deploy a Ruby on Rails application, you might have been greeted with a crash:

 => ERROR [linux/arm64 build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile                                                                     0.3s
------
 > [linux/arm64 build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile:
0.287 /usr/local/bundle/ruby/3.3.0/gems/concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/atomic/lock_local_var.rb:14: [BUG] Segmentation fault at 0x0039ffffa97e06c0
0.287 ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux]
0.287
0.287 -- Control frame information -----------------------------------------------
0.287 c:0096 p:---- s:0524 e:000523 CFUNC  :resume
...
0.290 Segmentation fault
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:38
--------------------
  36 |
  37 |     # Precompiling assets for production without requiring secret RAILS_MASTER_KEY
  38 | >>> RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
  39 |
  40 |
--------------------
ERROR: failed to solve: process "/bin/sh -c SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile" did not complete successfully: exit code: 139

There is a bug in Ruby 3.3.0, which has been addressed, but a new Ruby patch version has not been released yet. If you have already pulled the image from the registry, you can either update the image and continue using the tag, or you can specify the manifest digest of the recently rebuilt image:

# Make sure Ruby version matches .ruby-version
#   manifest-tool inspect registry.docker.com/library/ruby:3.3.0-slim
FROM registry.docker.com/library/ruby@sha256:abcb7c3943a085f511397b65ba7ca32ad56af759a53af932c5e354e1a8f84bcb AS base

Change history