Build a multi-architecture docker image with Vagrant

8 min readMar 17, 2020

As a developer, you want a consistent environment to do repeated task and have a predictable output of your program. In this case, you may want to consider adding vagrant into your deployment workflow.

Before you jump straight into deployment with Vagrant, you will need a provider. Here I will use virtual box. The reason is simple: free.

As Vagrant official website state: Vagrant comes with support out of the box for VirtualBox, a free, cross-platform consumer virtualisation product.

Installation Guide

There are multiple articles already tell you how to install them.

An introduction to Vagrant

What is vagrant?

medium.com

The Ultimate Guide to Getting Started with Vagrant and Why Every Software Developer Should’ve Been…

Why do all the blog posts ever make vagrant into rocket science?

medium.com

I am going to use Virtualbox and Vagrant to setup RHEL8 on macOS Catalina 10.15.3.

Objective

My goal is to allow me to run a RHEL server which can build multi architecture docker image and allow you to pass in environment variable for customisation.

To build multi architecture, traditionally you are required hardware support. For example, you will need to buy a power server just to build container image that run smoothly in power environment which is not really cost effective.

Or maybe you have raspberry-pi stack, you want to run docker image on that, and realised that you faced exec format error.

Lucky enough, we have docker buildx in 2020! With the recent introduction of Docker’s buildx functionality, you can now easily to build multi-arch container image!

Prerequisite

REDHAT DEVELOPER ACCOUNT

Register an account here.
If you encountered password error, probably your password has special character. In this case, you may want to change the password or using the activation key.
if you want to use activation key, can try this https://access.redhat.com/solutions/3341191 , https://access.redhat.com/articles/1378093

2. DOCKER HUB ACCOUNT

3. Git

Getting Stated with this project

Ensure you have properly set up your environment in Installation Guide section.

You can get the source code here

$ git clone https://github.com/DanielChuDC/multiarch-vagrant-rhel
$ cd multiarch-vagrant-rhel# Check up the files 
$ tree ..
|-- LICENSE
|-- Readme.md
|-- Vagrantfile
|-- example.env
|-- golang-deployment.yaml
`-- provisioner.sh0 directories, 6 files

Vagrantfile content

# -*- mode: ruby -*-
# vi: set ft=ruby :# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.# Every Vagrant development environment requires a box. You can search for
# boxes at https://vagrantcloud.com/search.
VM_BOX  =  "generic/rhel8"Vagrant.configure(2) do |config|
  config.env.enable # enable the plugin
  config.vm.box = VM_BOX
  config.ssh.extra_args = ["-t", "cd /home/vagrant; bash --login"] #https://stackoverflow.com/questions/17864047/automatically-chdir-to-vagrant-directory-upon-vagrant-ssh
  config.vagrant.plugins = "vagrant-env"
  config.vm.provider "virtualbox" do |vb|
    vb.memory = 4096 
    vb.cpus = 4
  end
  config.vm.provision 'shell' do |s| 
     s.path = 'provisioner.sh'
     s.env = { "RH_ACCOUNT_USERNAME"=>ENV['RH_ACCOUNT_USERNAME'], 
     "RH_ACCOUNT_PASSWORD"=>ENV['RH_ACCOUNT_PASSWORD'], 
     "ORG_ID"=>ENV['ORG_ID'], 
     "Key_NAME"=>ENV['Key_NAME'], 
     "DOCKER_HUB_USERNAME"=>ENV['DOCKER_HUB_USERNAME'], 
     "DOCKER_HUB_PASSWORD"=>ENV['DOCKER_HUB_PASSWORD'], 
     "DOCKER_IMAGE_NAME"=>ENV['DOCKER_IMAGE_NAME'], 
     "DOCKER_IMAGE_TAG"=>ENV['DOCKER_IMAGE_TAG'], 
     "GIT_REPO_URL"=>ENV['GIT_REPO_URL'],
     "GIT_BRANCH"=>ENV['GIT_BRANCH'],
     "TARGET_PLATFORM"=>ENV['TARGET_PLATFORM']}
  endconfig.trigger.before :destroy do |trigger|
    trigger.warn = "Unregister redhat developer account"
    trigger.run_remote = {
      inline: "
      if subscription-manager status; then 
      sudo subscription-manager unregister 
      fi
      "}
  end
end

provisioner.sh content

#! /bin/bash   
set -mecho "Setting environment variable"echo $RH_ACCOUNT_USERNAME
echo $RH_ACCOUNT_PASSWORD
echo $ORG_ID
echo $Key_NAME
echo $DOCKER_HUB_USERNAME
echo $DOCKER_HUB_PASSWORD
echo $DOCKER_IMAGE_NAME
echo $DOCKER_IMAGE_TAG
echo $GIT_REPO_URL
echo $GIT_BRANCH # Add in git branch feature 
echo $TARGET_PLATFORMecho "Checking this OS image and kernel version"
cat /etc/redhat-release
if ! subscription-manager status; then
sudo subscription-manager register --username=$RH_ACCOUNT_USERNAME --password=$RH_ACCOUNT_PASSWORD --auto-attach
sudo  subscription-manager register --org=$ORG_ID --activationkey=$Key_NAME
sudo subscription-manager attach
sudo subscription-manager list
fi
sudo yum update -y
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
echo "Cannot install directly, faced error while install containerd.io"
echo "current work around is to manually install containerd.io"
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf repolist -v
sudo dnf list docker-ce --showduplicates | sort -r
sudo dnf install docker-ce-3:19.03.8-3.el7
sudo dnf install --nobest docker-ce -y
sudo dnf install https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm -y
sudo systemctl enable docker
sudo systemctl start dockerecho "validate docker running successfully"
sudo docker run hello-worldecho "groupadd docker user, so we don't run permission with sudo"
sudo groupadd docker
sudo usermod -aG docker vagrantecho "getting root permission"
export DOCKER_CLI_EXPERIMENTAL=enabledecho "Install qemu for RHEL 8"
sudo yum install qemu-kvm -y
echo "Checking qemu version"
sudo /usr/libexec/qemu-kvm --version
sudo yum info qemu-kvmecho "Checking Kernel Version"
uname -recho "login docker "
sudo docker login -u $DOCKER_HUB_USERNAME --password=$DOCKER_HUB_PASSWORDecho "Install jq "
sudo yum install jq -y# Using jq to persist the buildx changes
sudo jq '. + {"experimental": "enabled"}' /root/.docker/config.json > abc.json
cat abc.json
sudo mv abc.json /root/.docker/config.json# Add dns record for daemon.json
echo '{"dns":["8.8.8.8","8.8.4.4"]}' | sudo tee -a /etc/docker/daemon.jsonecho "stop RHEL firewall"
sudo systemctl stop firewalldecho "make docker bridge able to access internet"
sudo sysctl net.ipv4.conf.all.forwarding=1
sudo iptables-save > your-current-iptables.rules
sudo iptables --flush
sudo iptables -P FORWARD ACCEPT
sudo iptables -I INPUT -j ACCEPT
sudo iptables -I OUTPUT -j ACCEPTecho "Restart docker to make effect"
sudo systemctl restart dockerecho "Enable DOCKER_CLI_EXPERIMENTAL Flag"
export DOCKER_CLI_EXPERIMENTAL=enabled
echo 'export DOCKER_CLI_EXPERIMENTAL=enabled' > .bash_profile
source .bash_profileecho "validate buildx command"
echo $DOCKER_CLI_EXPERIMENTAL
docker buildx buildecho "Enable binfmt_misc to run non-native Docker images"
sudo docker run --restart always --privileged multiarch/qemu-user-static --reset -p yesecho "Checking binfmt_misc"
ls /proc/sys/fs/binfmt_misc/
cat /proc/sys/fs/binfmt_misc/qemu-aarch64echo "Creating a Buildx Builder"
docker buildx create --use --name mybuilder
docker buildx inspect --bootstrap
docker buildx lsecho "Installing development tools in RHEL 8"
sudo yum install git -yecho "Git clone the url"
if [ -d "example" ]; then
   echo "folder example exists. Delete it now."
  rm -rf ./example
fi
if [ -z "$GIT_BRANCH" ]; then
   echo "\$GIT_BRANCH is NULL set to default"
   export GIT_BRANCH='master'
else
      echo "\$GIT_BRANCH is NOT NULL"
fi
git clone $GIT_REPO_URL ./example 
cd /home/vagrant/example && git branch -a # list your repo
cd /home/vagrant/example && git checkout $GIT_BRANCH # Change branch
cd /home/vagrant/example && git branch # Confirm you are now working on that branchecho "Build Multiarch Image using Dockerfile and Buildx"
cd /home/vagrant/example && docker buildx build -t $DOCKER_HUB_USERNAME/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG --platform=$TARGET_PLATFORM . --push

Customisation with environment variable

If you are a developer yet not familiar with environment variable, there is a introduction article here to show how important it is.

An Introduction to Environment Variables and How to Use Them

Decoupling configuration from the application

medium.com

$ cat example.envRH_ACCOUNT_USERNAME=<your-rh-account-username>
RH_ACCOUNT_PASSWORD=<your-rh-account-password>
ORG_ID=<your-org_id>  # optional,can left blank,unless your password has special character
Key_NAME=<your-keyname> # optional,can left ,unless your password has special character 
DOCKER_HUB_USERNAME=<your-docker-hub-username>
DOCKER_HUB_PASSWORD=<your-docker-hub-password>
DOCKER_IMAGE_NAME=<your-image-name>
DOCKER_IMAGE_TAG=<your-image-tag>
GIT_REPO_URL=https://github.com/DanielChuDC/loopback-example #change as you wish
GIT_BRANCH=master
TARGET_PLATFORM=linux/arm64,linux/amd64,linux/s390x,linux/ppc64le
# target platform
# possible values: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Rename example.env to .env and fill in the correct credential accordingly.

You should not check in .env that contains sensitive information/ credentials in Github.

Here we want to pass in the environment variable to the Vagrantfile using vagrant plugin, called vagrant-env .

$ vagrant plugin install vagrant-env

Optionally, you can use export function without using .envto pass in variable.

export RH_ACCOUNT_USERNAME='<your-rh-account-username>';
export RH_ACCOUNT_PASSWORD='<your-rh-account-password>';
export ORG_ID='<your-org_id>';
export Key_NAME='<your-keyname>';
export DOCKER_HUB_USERNAME='<your-docker-hub-username>';
export DOCKER_HUB_PASSWORD='<your-docker-hub-password>';
export DOCKER_IMAGE_NAME='<your-image-name>';
export DOCKER_IMAGE_TAG='<your-image-tag>';
export GIT_REPO_URL='https://github.com/DanielChuDC/loopback-example';
export TARGET_PLATFORM='linux/arm64,linux/amd64,linux/s390x,linux/ppc64le';

Ready to bake your first multi-arch container image

Provision your RHEL server and bake your container image by using

# start server
$ vagrant up

In this project, I have set up an example Github repo by using Loopback.

Waiting for it to download and install all the softwares and push your multi-arch container image into docker hub

If you see the similar message about exporting image and pushing image successful, means it done!

Check the image status on docker hub

You should go to your repo to check if you have pushed your image.

Validation: In zLinux

get your instance of zLinux from IBM LinuxONE Community Cloud, once you have a VM server running, ssh into it and run the following commands. Do refer to Part 1 and Part 2 on how to use LinuxONE.

$ docker run --rm -d -p 3000:3000 moxing9876/multi-arch-vagrant:3.0.3
# expected output
Status: Downloaded newer image for moxing9876/multi-arch-vagrant:3.0.3
26e3475d3a3f697c1c5d655df45a7693b49d50286272efc0c0326538eaa843b5$ curl http://localhost:3000/ping
# expected output
{"greeting":"Hello from LoopBack","date":"2020-03-17T14:52:27.320Z","url":"/ping","headers":{"user-agent":"curl/7.29.0","host":"localhost:3000","accept":"*/*"}}$ $ docker image inspect moxing9876/multi-arch-vagrant:3.0.3 | grep Architecture
# expected output
"Architecture": "s390x",

Clean up

When you do not need this RHEL server and want to delete this project, run this command in the terminal

# Run when you want to delete this server
$ vagrant destroy -f #destroy server
# expected output
==> default: Running action triggers before destroy ...
==> default: Running trigger...
==> default: Unregister redhat developer account
    default: Running: inline script
    default: Unregistering from: subscription.rhsm.redhat.com:443/subscription
    default: System has been unregistered.
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...

Summary

This project show the ability of building multi-arch container image by vagrant, virtual box, buildx, qemu. It can be a proof-of-concept for CI/CD workflow of building multi-arch container image. Not only it save cost, it enable better documentation and consistent development environment.

You may want to host this on IBM Cloud, AWS, Google Cloud by using terraform.

p/s: Thank you Jaric Sng, show me how important environment variable is, and guide me on this project

Do check out his 4 articles on

Part 1: Multi-architecture cloud native Loopback (nodejs) in LinuxONE and others
Part 2: Multi-architecture cloud native Loopback (nodejs) in LinuxONE and others
Part 3: Multi-architecture cloud native Java microservice for LinuxONE and others
Part 4: Creating a build server for Cloud native Multi architecture docker build (Part 4)

Build a multi-architecture docker image with Vagrant

Installation Guide

An introduction to Vagrant

What is vagrant?

The Ultimate Guide to Getting Started with Vagrant and Why Every Software Developer Should’ve Been…

Why do all the blog posts ever make vagrant into rocket science?

Objective

Prerequisite

Getting Stated with this project

Vagrantfile content

provisioner.sh content

Customisation with environment variable

An Introduction to Environment Variables and How to Use Them

Decoupling configuration from the application

Ready to bake your first multi-arch container image

Validation: In zLinux

Clean up

Summary

See you next time!

Written by Daniel Chu