Our Learnings on Real-world Robotics Docker Workflows

In a previous post, we discussed a potential development and deployment workflow that uses development containers (devcontainers) as the primary unit of software encapsulation. In this approach, the development environment is encoded and tracked by a devcontainer. That container image is, in turn, reused as a deployment medium in the robot.

In this post, we share some of the lessons we’ve learned while working with this system.  We hope our challenges, successes, and the insights gained along the way can help others considering a similar approach.

‍

Lesson #1: Treat Docker images as production software

When used as a build system, Docker images have implicit requirements. They need to support building the entire codebase, running all tests, and running any standalone application.

Thus, one should embrace Docker image implementations as any other software development process:

• survey and document requirements,
• manage changes over them,
• aim at fulfilling them one at a time with incremental improvements

If you’re replacing an existing build system, avoid breaking changes until the new implementation is fully functional. Hot swaps rarely go as smoothly as expected.

‍

Lesson #2: Easier pulled than built

Dockerfiles are easy to manage: they are lightweight text files that can be easily included in our repositories. Anyone with access to them and the Docker software can turn them into a working container, right?

In practice, we have found that this is not always the case.

Docker builds take time and space (more on that later). They are also brittle, as they tend to depend on internet services, e.g. mirrors and indexes, being available and not banning you (I’m looking at you, raw.githubusercontent).

Not to mention end users with Windows and WSL setups…

If you rely on each end user building their own images, you will still maintain as many build configurations as you have developers, which defeats the purpose of using a devcontainer in the first place.

At the end of the day, Docker images are software too, and they're usually better served already compiled.

We want to build our container images in a centralized process and serve them to end users ready to use. A CI that builds the Docker image and pushes it to a registry is a good example. Users can simply download the latest precompiled image as a nicely wrapped gift.

‍

Lesson #3: Building large containers takes more space than you think

In robotics, Docker images can grow fast. ROS can take a couple of gigabytes, LLMs can take ten or more, and CUDA libraries add a few more on top. You blink, and suddenly you're dealing with a thirty-gigabyte Docker image.

Building such an image can require twice as much disk space as the final artifact. This is due to intermediate structures being kept around.

Disk space is relatively cheap these days, but you won’t find 60GB of available storage in most low-tier cloud runners, including GitHub action runners.

The takeaway sounds obvious, but might save you hours of despair: if your image is larger than ~10GB, shoot for a high-tier cloud runner or a local one directly.

‍

Lesson #4: Hitting Docker image cache can be a fine art

Docker image files specify a layer on each line. Building them involves applying each layer on top of the previous one. To avoid repeatedly executing this time-consuming process, Docker implements a layer cache. If nothing has changed in a given layer specification (a Dockerfile line) or in the previous ones, the cache is used instead of rebuilding.

Cache is stored as a shelf of precompiled intermediate layers that can be used at any time during builds.

The first corollary is: when iterating on a Dockerfile, add new lines as late as possible to reuse cached layers during your tests.

A related corollary: COPY instructions invalidate the cache for every layer that follows if any file in the copied path changes, even files you don't care about. Keep a tight .dockerignore to exclude build artifacts, editor files, and local state from the build context. Beyond preventing spurious cache misses, this also reduces context transfer time and avoids accidentally baking sensitive files into the image.

If you are building images in the cloud as part of PR validations, caching becomes important to lower CI lag and costs. Caching in cloud environments can be tricky, because one gets a fresh instance for every build, so caching must be stored outside the builders.

There are several options for this. GitHub provides a caching mechanism, known as gha, which appears to be the best option at first due to its transparent integration with GitHub workflows. However, cache tenancy policies and limits make it less reliable and predictable than one would like.

We have found Docker inlined cache to be the simplest and most reliable form of caching. This approach adds caching metadata to the images already pushed to the registry so that they can be used as a cache source.

To maximize cache hits on a CI, we recommend pushing the built image tagged with the branch name. Then, both that tag (repeated builds on the same PR) and the main branch tag (for the first build) can be used as cache sources, maximizing cache hits across CI builds.

‍

Lesson #5: Cache mounts speed up local rebuilds

For images that install heavy dependencies, pip packages, Conan C++ libraries,  every Dockerfile edit can trigger a full reinstall, even when the dependency list hasn't changed. BuildKit cache mounts solve this: unlike the layer cache, they persist on the host between builds and survive changes in unrelated layers.

RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

RUN --mount=type=cache,target=/root/.conan conan install.

Cache mounts are a local developer tool, not a CI cache. On CI, each runner starts with a fresh filesystem, so cache mounts are always empty. The inline cache strategy described above remains the right answer there.

‍

Conclusions

Working with Docker images in real-world robotics applications takes more skill than it may seem at first glance. In this post, we shared a few of the lessons we’ve learned along the way, and we hope you found them useful.

At Focus, we're constantly refining these practices across our robotics projects, and we'd love to hear your take as well. Feel free to reach out and continue the conversation.

Contributors: Javier Cremona, Martin Llofriu, Marco Rolón

‍