Automating Image Updates with FluxCD on AKS

2023-09-22 13 min read GitOps Kubernetes Developer Tutorial

In my previous post, we walked through the setup of FluxCD on AKS via AKS extensions. In this article, we’ll go a bit deeper and take a look at how you can use FluxCD to automate image updates in your AKS cluster.

The goal here is to streamline the process of updating your application deployments in your cluster.

Here is our intended workflow:

Modify application code, then commit and push the change to the repo.
Create a new release in GitHub which kicks off a release workflow to build and push an updated container image to a GitHub Container Registry.
FluxCD detects the new image and updates the image tag in the cluster.
FluxCD rolls out the new image to the cluster.

We’ll use same AKS store demo app we used in the previous post, but this time we’ll go a bit faster.

If you need a refresher on how to deploy the application, you can refer to the previous post.

Prerequisites

Before you begin, you need to have the following:

Create an AKS cluster

Use the following Azure CLI commands to create a new resource group and AKS cluster.

RG_NAME=rg-imageupdate
AKS_NAME=aks-imageupdate
LOC_NAME=westus3

az group create -n $RG_NAME -l $LOC_NAME
az aks create -n $AKS_NAME -g $RG_NAME

When the cluster is created, run the following command to connect to the cluster.

az aks get-credentials -n $AKS_NAME -g $RG_NAME

Bootstrapping FluxCD

In my previous post, I used the AKS extension to install FluxCD. This time, I’ll be using the Flux CLI to bootstrap FluxCD onto the AKS cluster. This process will enable us to have a bit more control over the installation and gives us the ability to save the Flux resources to our GitHub repo.

So we’re going to implement GitOps even on top of our GitOps tooling 🤯

There’s a few things we need to do before we can bootstrap the cluster. First, we need the Flux CLI. Once you have the CLI installed, run the following command to ensure your cluster is ready for bootstrapping.

flux check --pre

We also need a repo to land the Flux resources into. So let’s fork and clone the aks-store-demo-manifests repository. This with be our repo for both application deployment and our GitOps configurations.

We’ll be doing a bunch of things within GitHub. I prefer using the GitHub CLI over clicking around in the GitHub website.

Run the following commands to login to GitHub, clone the repo and prep it for our exercise.

# navigate to the home directory or wherever you want to clone the repo
cd ~

# login to GitHub
gh auth login --scopes repo,workflow

# fork and clone the repo to your local machine
gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone

# make sure you are in the aks-store-demo-manifests directory
cd aks-store-demo-manifests

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default

# merge the "kustomize" branch into "main"
git merge origin/kustomize

# push the change up to your fork
git push

We need to set some environment variables to for the Flux bootstrapping process. Run the following commands to set the environment variables.

# set the repo url
export GITHUB_REPO_URL=$(gh repo view --json url | jq .url -r)

# set your GitHub username
export GITHUB_USER=$(gh api user --jq .login)

# set your GitHub personal access token
export GITHUB_TOKEN=$(gh auth token)

Now we’re ready to bootstrap FluxCD. Run the following command to bootstrap FluxCD. This command will install FluxCD with additional components to enable image automation. It will also generate the Flux resources and commit them to our Git repo in the clusters/dev directory.

flux bootstrap github create \
  --owner=$GITHUB_USER \
  --repository=aks-store-demo-manifests \
  --personal \
  --path=./clusters/dev \
  --branch=main \
  --reconcile \
  --network-policy \
  --components-extra=image-reflector-controller,image-automation-controller

After a few minutes, run the following commands and you should see the Flux resources in the repo.

git fetch
git rebase

# show the Flux resources
tree clusters/dev

In order for the image-automation-controller to write commits to our repo, we need to create a Kubernetes secret to store our GitHub credentials. Run the following command to create the secret.

flux create secret git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN

This command will create Kubernetes secrets in your cluster but you could also use Azure Key Vault with the Secret Store CSI driver to store your GitHub credentials.

We don’t need the GitHub PAT token anymore, so run the following command to discard it.

unset GITHUB_TOKEN

Next we need to create a GitRepository resource. This resource will point to our fork of the aks-store-demo-manifests repo where we have our app deployment and GitOps manifests stored.

Run the following command to create the configuration and export it to a YAML file which we’ll commit to our repo.

flux create source git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --branch=main \
  --interval=1m \
  --secret-ref=aks-store-demo \
  --export > ./clusters/dev/aks-store-demo-source.yaml

We also need to specify the Kustomization resource to tell FluxCD where to find the app deployment manifests in our repo. Run the following command to create the configuration and export it to a YAML file which we’ll also commit to our repo.

flux create kustomization aks-store-demo \
  --source=aks-store-demo \
  --path="./overlays/dev" \
  --prune=true \
  --wait=true \
  --interval=1m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/dev/aks-store-demo-kustomization.yaml

We have two new Flux resource manifests. Run the following command to commit and push the files to the repo so Flux can begin the reconciliation process.

git add -A
git commit -m "feat: add source and kustomization" 
git push

As soon as the code is pushed, Flux will do it’s thing; reconcile. Run the following command to watch the Kustomization reconciliation process.

flux get kustomizations --watch

After a few minutes you should see something like this…

aks-store-demo	main@sha1:95de7bde	False	True	Applied revision: main@sha1:95de7bde

Confirm the GitRepository and Kustomization resources have been created.

flux get sources git 
flux get kustomizations

Run the following command to ensure all the Pods are running.

kubectl get po -n dev

Once you see all the Pods are running, run the following command to grab the public IP of the store-front service.

kubectl get svc store-front -n dev

Using a web browser, navigate to the public IP and confirm the application is up and running.

Setup the release workflow

The source code for the aks-store-demo application is located in a different repository than where our manifests are.

To give you a bit of a warning, we’ll be flipping back and forth between the aks-store-demo and aks-store-demo-manifests repository directories.

Hint: you can use the cd - command to flip back and forth between the last two directories you were in.

The application code is in the aks-store-demo repository. Run the following commands to fork and clone the repo.

# navigate to the home directory or wherever you want to clone the repo
cd -

# fork and clone the repo to your local machine
gh repo fork https://github.com/azure-samples/aks-store-demo.git --clone

# make sure you are in the aks-store-demo directory
cd aks-store-demo

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default

Currently the aks-store-demo repository has Continuous Integration (CI) workflows to build and push container images using GITHUB_SHA as the image tag. We need to create a new release workflow that will build and push a new container image using semantic versioning. Flux’s image update automation policy which is used to determine the most recent image tag can use various methods including numerical tags, alphabetical tags, and semver tags. So in this case, as much as I’d like to use the GITHUB_SHA, it does not provide a conducive way to determine “latest”.

Therefore, we need to tag our releases with semantic version numbers. GitHub offers a way to create tagged releases. Additionaly, GitHub Actions have the ability to trigger workflows when a new release is published.

Run the following command to create a new workflow file.

# make sure you are in the aks-store-demo directory
touch .github/workflows/release-store-front.yaml

Open the release-store-front.yaml file using your favorite editor and paste the following code.

name: release-store-front

on:
  release:
    types: [published]

permissions:
  contents: read
  packages: write

jobs:
  publish-container-image:
  
    runs-on: ubuntu-latest

    steps:
      - name: Set environment variables
        id: set-variables
        run: |
          echo "REPOSITORY=ghcr.io/$(echo ${{ github.repository }} | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_OUTPUT"
          echo "IMAGE=store-front" >> "$GITHUB_OUTPUT"
          echo "VERSION=$(echo ${GITHUB_REF#refs/tags/})" >> "$GITHUB_OUTPUT"
          echo "CREATED=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT"          

      - name: Env variable output
        id: test-variables
        run: |
          echo ${{ steps.set-variables.outputs.REPOSITORY }}
          echo ${{ steps.set-variables.outputs.IMAGE }}
          echo ${{ steps.set-variables.outputs.VERSION }}
          echo ${{ steps.set-variables.outputs.CREATED }}          

      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ github.token }}

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: src/store-front
          file: src/store-front/Dockerfile
          push: true
          tags: |
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:latest
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:${{ steps.set-variables.outputs.VERSION }}            
          labels: |
            org.opencontainers.image.source=${{ github.repositoryUrl }}
            org.opencontainers.image.created=${{ steps.set-variables.outputs.CREATED }}
            org.opencontainers.image.revision=${{ steps.set-variables.outputs.VERSION }}

This workflow will be triggered each time a new release is published and the ${GITHUB_REF#refs/tags/} bit allows us to extract the release version to use as an image tag.

Save the file, then commit and push the changes to the repo.

git add .github/workflows/release-store-front.yaml
git commit -m "feat: add release workflow"
git push

Update the application

Since we’re in the app repo, let’s make a change to it.

We’ll make a very small edit to the src/store-front/src/components/TopNav.vue file and update the title of the application to include a version number.

Open the file and change the title on line 4 from Azure Pet Supplies to Azure Pet Supplies v1.0.0.

If you don’t want to use an editor, you can run the following command to make the change using sed (I love and hate sed at the same time).

# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies/Azure Pet Supplies v1.0.0/g" src/store-front/src/components/TopNav.vue

Commit and push the changes to the repo.

git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title"
git push

Using GitHub CLI, create and publish a new release tagged as 1.0.0. This will trigger the release workflow we just created.

gh release create 1.0.0 --generate-notes

Wait about 10 seconds then run the following command to watch the workflow run.

# you can select the release workflow from the list
gh run watch

With the store-front container image release workflow setup, let’s configure image update automation in Flux.

Configure image update automation

To setup FluxCD to listen for changes in the GitHub Container Registry, we need to create a new ImageRepository resource. You need to create an ImageRepository resource for each image you want to automate. In this case, we’re only automating the store-front image.

⚠️ Flip back over to the aks-store-demo-manifests repo

Using the FluxCLI, run the following command to create the manifest for the ImageRepository resource.

flux create image repository store-front \
  --image=ghcr.io/$GITHUB_USER/aks-store-demo/store-front \
  --interval=1m \
  --export > ./clusters/dev/aks-store-demo-store-front-image.yaml

Run the following command to create an ImagePolicy resource to tell FluxCD how to determine the newest image tags. We’ll use the semver filter to only allow image tags that are valid semantic versions and equal to or greater than 1.0.0.

flux create image policy store-front \
  --image-ref=store-front \
  --select-semver='>=1.0.0' \
  --export > ./clusters/dev/aks-store-demo-store-front-image-policy.yaml

There are other filters you can use as well. You can read more about them here.

Finally, run the following command to create an ImageUpdateAutomation resource which enables FluxCD to update images tags in our YAML manifests.

flux create image update store-front \
  --interval=1m \
  --git-repo-ref=aks-store-demo \
  --git-repo-path="./base" \
  --checkout-branch=main \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --commit-template="{{range .Updated.Images}}{{println .}}{{end}}" \
  --export > ./clusters/dev/aks-store-demo-store-front-image-update.yaml

If you are uncomfortable with FluxCD making changes directly to the checkout branch (in this case main), you can create a separate branch for FluxCD (using the --push-branch parameter) to specify where commits should be pushed to. This will then enable you to follow your normal Git workflows (e.g., create a pull request to merge the changes into the main branch).

Run the following commands to commit and push the new Flux resources to the repo.

git add -A
git commit -m "feat: add image automation"
git push

After a few minutes, run the following command to see the status of the image update automation resources.

flux get image repository store-front
flux get image policy store-front
flux get image update store-front

The image update automation is setup but it won’t do anything until we tell it which Deployments to update.

Update the manifest

We’re going to sick with updating the store-front only. So we’ll need to update the store-front deployment manifest to use the ImagePolicy resource we created earlier. This is done by marking the manifest with a comment.

Open the base/store-front.yaml file using your favorite editor.

On line 19, update the image to use your GitHub Container Registry. Then at the end of the line, add a comment to include the namespace and name of your ImagePolicy resource. This marks the deployment for Flux to update.

The comment is super important because Flux will not implement the image tag policy if it doesn’t have this comment.

The line should look something like this:

image: ghcr.io/<REPLACE_THIS_WITH_YOUR_GITHUB_USERNAME>/aks-store-demo/store-front:latest # {"$imagepolicy": "flux-system:store-front"}

Setting the marker in the deployment manifest is fine for this demo, but ideally you’d want to set it in the kustomization manifest instead.

Commit and push the changes to the repo.

git add base/store-front.yaml
git commit -m "feat: add imagepolicy to store-front manifest"
git push

At this point, we’ve successfully setup image automation in our cluster. Time to test.

Test the image automation

We’re going to test the developer workflow we laid out at the beginning of this article.

Make another change

Flip back over to the aks-store-demo repo

Make another change to the TopNav.vue file. This time, change the title to Azure Pet Supplies v2.0.0.

# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies v1.0.0/Azure Pet Supplies v2.0.0/g" src/store-front/src/components/TopNav.vue

Commit and push the changes to the repo.

git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title again"
git push

Create a new 2.0.0 release. This will trigger the release workflow.

gh release create 2.0.0 --generate-notes

# wait about 5 seconds then run the following command
gh run watch

Verify the image update

After a few minutes, you should see the new image tag in the aks-store-demo repo and the store-front deployment being updated in the cluster.

The reconcile interval for ImagePolicy was set to 1 minute. So after 1 minute or so, we should see that the update was successful.

flux get image policy store-front

Now check the store-front deployment to see the new image tag.

kubectl get deploy store-front -n dev -o yaml | grep image:

You should see that the image tag has been updated to 2.0.0.

Sweet! We’ve successfully automated image updates in our cluster using FluxCD and GitOps 🚀

Get the public IP of the store-front service by running the following command.

kubectl get svc store-front -n dev

Using a web browser, navigate to the public IP address of the store-front service. You should see the application running with our new title that includes v2.0.0 🥳

Conclusion

The power of GitOps is on full display here 🤖

In this article, we took a look at how you can use leverage an innovative feature of FluxCD to automate image updates in your AKS cluster. As an app developer, I can simply make my code changes and push it through a PR process. Once the PR is approved and merged and a new release is created, the image update automation kicks in and updates the image tag in the cluster for us without any manual intervention. This is a great way to streamline the deployment process and keep your cluster up to date with the latest images.

As an aside, I did all this open source FluxCD. This could also be done using the AKS extension for FluxCD but you do need to enable the image-automation-controller and image-reflector-controller components. You can find more information on how to do that here).

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

Peace ✌️

PAUL'S BLOG

Automating Image Updates with FluxCD on AKS

Prerequisites

Create an AKS cluster

Bootstrapping FluxCD

Setup the release workflow

Update the application

Configure image update automation

Update the manifest

Test the image automation

Make another change

Verify the image update

Conclusion

Resources