PAUL'S BLOG

Learn. Build. Share. Repeat.

Building Multi-Architecture Container Images

2023-09-05 10 min read Containerization Multiple Architecture Docker DevOps Container Images Developer Tips

Introduction

Over the past several years, we’ve seen the emergence of the ARM64 architecture, which is gaining popularity due to its energy efficiency and performance benefits. We often seen these processors in mobile devices, such as smartphones and tablets. We also seen them in our IoT devices, such as smartwatches and smart TVs. Now, we are starting to see increased adoption of ARM64 processors in servers and personal computers.

This emergence of the ARM64 architecture, means that we now have two major processor architectures to consider when building and deploying our containerized applications in order to maximize our application’s reach.

In this article, we’ll explore the importance of multi-architecture (multi-arch) container images, how to build them, and how to deploy them on Azure Kubernetes Service (AKS).

Why multi-arch container images matter

When distributing a containerized application, broad compatibility across different platforms is essential for widespread adoption. Whether you’re sharing your container image with your team or the wider community, ensuring it runs seamlessly on various architectures is critical.

When it comes to cloud computing, cloud providers offer a wide range of processor architectures, each with its own set of benefits and drawbacks. For example, we know that ARM64 processors are more energy efficient than x86_64 processors. This means that running your containerized application on an ARM64 processor will consume less power than running it on an x86_64 processor. This can lead to significant cost savings for your organization.

By creating multi-arch container images, you create a single container image that contains multiple versions of your application, each targeting a specific OS architecture. This allows the client OS to determine the appropriate architecture to utilize.

Acronyms galore

Before we dive into building multi-arch container images, it is important to understand some of nomenclature used to identify OS processor architectures. When it comes to identifying OS processor architectures, it is full of acronyms and can be a bit dizzying to decipher and it can be challenging to keep track of all the different processor architectures.

Here’s what we need to know:

  • x86 is a family of instruction set architectures based on the Intel 8086 CPU. The is the most widely used architecture and is supported by most operating systems. x86_64 is a 64-bit version of the x86 architecture.
  • ARM is a family of instruction set architectures based on the ARM CPU. This is a newer architecture that is gaining popularity due to its energy efficiency and performance benefits. ARM64 is a 64-bit version of the ARM architecture and you might also see it identified as aarch64.

So if you see, x86_64 just know that its synonymous with AMD64, and aarch64 is synonymous with ARM64.

Approaches to build multi-arch container images

There are a few strategies for building container images that support multiple architectures. The first one is to build a separate image for each architecture. This approach has one huge drawback; you’ll need to maintain multiple images and keep them in sync. From an end-user perspective, this means that they will need to know which image to use for their specific host OS architecture.

Another approach is to build each container image separately, then combine them into a single image using the Docker manifest command. This ultimately gives you a multi-arch container image; however, this approach is cumbersome and error-prone.

And, do you really want to update the manifest file every time you build a new image?!? Probably not… there is a better way.

A better approach is to use a tool like Docker Buildx to build a single image which can run on multiple architectures. Docker Buildx is a tool that simplifies the process of building multi-arch container images. With this tool, you can use a single command to build multiple container images, each targeting a specific architecture, and combine the images into a single multi-arch image.

This is the way

Multi-arch container images in action

To demonstrate how to build multi-arch container images, we’ll use a simple Rust application that I wrote. You can find the source code for this application here. This application is a simple command line tool that prints out information about the OS it is running on.

Testing the application locally

I’ll start by cloning the repo and running the app locally to see what it does:

git clone https://github.com/pauldotyu/osinfo.git

cd osinfo

cargo run

You will need Rust installed on your machine to run this application. You can install Rust by following the instructions here.

I am running this M1 Mac (Apple Silicon) and this is what I see:

Type: Mac OS
Version: 13.5.1
Edition: None
Codename:
Bitness: 64-bit
Architecture: arm64

Testing the application on GitHub Codespaces

If you have access to GitHub Codespaces, open the repo in a Codespace and run the app.

You should see the following output which is different from what I saw on my M1 Mac:

Type: Debian
Version: 11
Edition: None
Codename: bullseye
Bitness: 64-bit
Architecture: x86_64

Running the application on AKS

What happens if I build a container from my M1 Mac, push it to Azure Container Registry (ACR), and attempt to run it on an AKS cluster?

Let’s find out…

I’ll need an AKS cluster, so I’ll use the Azure CLI to create one:

# Create a resource group
az group create \
  -n rg-multiarch \
  -l eastus

# Create an Azure Container Registry
ACR_NAME=$(az acr create -n acrmultiarch${RANDOM} \
  -g rg-multiarch \
  --sku Standard \
  --admin-enabled \
  --query name -o tsv)

# Create an AKS cluster
az aks create \
  -g rg-multiarch \
  -n aks-multiarch \
  --attach-acr $ACR_NAME

# Connect to the AKS cluster
az aks get-credentials \
  -g rg-multiarch \
  -n aks-multiarch

I am running these commands on my M1 Mac.

Run a container image built on ARM64 architecture

There is a Dockerfile in the repo that I can use to build a container image. I’ll use the docker build command to build and push the image from my M1 Mac:

docker build -t $ACR_NAME.azurecr.io/osinfo:v0.1-arm64 .
docker push $ACR_NAME.azurecr.io/osinfo:v0.1-arm64

When creating a Dockerfile that will be used to build multi-arch container images, you’ll need to ensure that the base image(s) supports the architectures you are targeting. For example, I am using is rust:1.72-bullseye and rust:1.72-slim-bullseye images and they both support linux/amd64 and linux/arm64 platforms. You can find more information about these image here and here

With the container pushed to ACR, run the container on the AKS cluster:

kubectl run osinfo1 --image=$ACR_NAME.azurecr.io/osinfo:v0.1-arm64 --restart=Never

If you run the command kubectl get po osinfo1, you’ll see that the pod is in a Error state.

NAME     READY   STATUS   RESTARTS   AGE
osinfo1  0/1     Error    0          7s

Run the command kubectl logs osinfo1, and you’ll see the following error message:

exec ./osinfo: exec format error

This means that the container image is not compatible with the architecture of the AKS node. The node is running on the x86_64 architecture and the container image is built for the ARM64 architecture. This is expected since I built the container image on my M1 Mac, which is running on the ARM64 architecture.

Run a container image built on AMD64 architecture

Let’s try building the container image on an x86_64 machine and see what happens. This time, I’ll use my GitHub Codespaces to build and push the container image by running the docker build and docker push commands as I did on my M1 Mac:

# Login to Azure
az login

# Get ACR name
ACR_NAME=$(az acr list -g rg-multiarch --query "[].name" -o tsv)

# Build and push container image
docker build -t $ACR_NAME.azurecr.io/osinfo:v0.1-amd64 .
docker push $ACR_NAME.azurecr.io/osinfo:v0.1-amd64

Run the container image on AKS:

# Get AKS credentials
az aks get-credentials --resource-group rg-multiarch --name aks-multiarch

# Run container image on AKS
kubectl run osinfo2 --image=$ACR_NAME.azurecr.io/osinfo:v0.1-amd64 --restart=Never

If you run the kubectl get po osinfo2 command, you should see the pod is in a Completed state.

NAME     READY   STATUS      RESTARTS   AGE
osinfo2  0/1     Completed   0          20s

If you run kubectl logs osinfo2, you’ll see the following output:

Type: Debian
Version: 11.0.0
Edition: None
Codename: 
Bitness: 64-bit
Architecture: x86_64

So we can see here that where the container image was built matters. If you build the container image on an x86_64 machine, it will run on an x86_64 machine. If you build the container image on an ARM64 machine, it will run on an ARM64 machine.

Build and run a multi-arch container image

Now, let’s use Docker Buildx to build a multi-arch container image. I’ll use the docker buildx command to build and push the image from my M1 Mac which has Docker Desktop installed. Buildx uses emulation to build container images for different architectures. This means that you can build container images for architectures that are different from the host machine.

If you are using Docker Desktop, Buildx should be enabled by default. You can verify this by running the following command:

docker buildx version

But to enable Buildx to build for multiple platforms, you’ll need to run the following command:

docker buildx create --use

When building container images using Buildx, you’ll need to know which “platform” the container image is targeting. Docker uses the combination of os/arch to identify the platform in which an image can run on. For example, linux/amd64 is the platform for running on Linux on x86_64 architecture and linux/arm64 is the platform for running on Linux on ARM64 architecture.

Run the following command to build the a multi-arch container image:

docker buildx build --platform linux/amd64,linux/arm64 --push -t $ACR_NAME.azurecr.io/osinfo:v0.1-multi .

Run the following command to confirm the container image supports multiple architectures:

docker manifest inspect $ACR_NAME.azurecr.io/osinfo:v0.1-multi

You should see output similar to the following and you will notice both AMD64 and ARM64 are included in the output. This means that your container image can run on either architecture.

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1055,
         "digest": "sha256:76781cd09ce46d5c0cf13b0c834c3d39e649c6c392c42343284ec851125f819b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1055,
         "digest": "sha256:26ebcb5ac14907804552477bf01f2892cdaebac06035e7561df4c5c61e6d8c30",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:875f61309b8199a0582faa4ebe8aaca9412e9982774a3bf9d9d24f5a61d7f020",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:76e5d0269adffa14b470b22856c653f466644b213b53d5fac0461fff88942362",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

AKS supports multiple node pools with different host OS architectures. We can tell Kubernetes to run a pod on a specific type of node by using a nodeSelector to ensure it runs on node that has the appropriate architecture.

Run the following command to run the container image on a node that has the x86_64 architecture:

kubectl run osinfo3 --image=$ACR_NAME.azurecr.io/osinfo:v0.1-multi --restart=Never --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"kubernetes.io/arch":"amd64"}}}'

Run kubectl get po osinfo3 and the kubectl logs osinfo3 commands, you’ll see the output is the same as the previous example. It ran successfully on the x86_64 node.

Now, let’s add a new ARM64 node pool to our AKS cluster, and see what happens when we run the container image on the new node pool.

az aks nodepool add \
  -g rg-multiarch \
  --cluster-name aks-multiarch \
  --name armpool \
  --node-vm-size Standard_B2ps_v2

Run the container image and use a nodeSelector again to ensure it runs on an ARM64 node:

kubectl run osinfo4 --image=$ACR_NAME.azurecr.io/osinfo:v0.1-multi --restart=Never --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"kubernetes.io/arch":"arm64"}}}'

If you run the kubectl logs osinfo4 command, you should see the following output:

Type: Debian
Version: 11.0.0
Edition: None
Codename: 
Bitness: 64-bit
Architecture: aarch64

This means that we’ve successfully built a multi-arch container image that can run on both x86_64 and ARM64 architectures 🎉

Conclusion

In this article, we explored the importance of multi-arch container images and how to build them using the Docker Buildx tool. Docker Buildx is valuable tool that simplifies the process of building multi-arch container images. By creating a single container image that supports multiple architectures, you can ensure that your application can run seamlessly on diverse hosts. This enables you to reach a wider audience, increase the adoption of your application, and potentially save some money in the cloud.

The knowledge we have gained here on how to use Docker Buildx will help us as we take a step further into automation. Because, running these docker buildx build commands locally is not ideal either.

So stay tuned for an upcoming article where I will show you how to build and publish multi-arch container images to Azure Container Registry using GitHub Actions and the Build and push Docker images GitHub Action 🚀

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

Peace ✌️

References

multi-architecture container docker-buildx cross-platform container-architecture
© 2024 by Paul Yu
Bilberry Hugo Theme