Custom images¶

This guide is intended primarily for

• System administrators who are trying to automate image creation for their labs
• Advanced users who have compelling reasons for using non-Ubuntu flavours

As mentioned in the warning above, the guide expects advanced understanding of operating systems and shell programming. We may not be able to provide any further support especially when it comes to non-Ubuntu/Debian based systems.

Environment¶

For creating custom images, we will be using a regular VM but the process can be easily adapted to run on your laptop. We will not be building anything locally, so a modest 2cpu-8ram-hpcv3 with uzh-only network interface will suffice.

To keep everything tidy, let's create a separate directory.

mkdir packer && cd packer

We will need the following additional software.

1. OpenStack python client that we will install into a virtual environment.

   python -m venv venv
   source venv/bin/activate
   pip install python-openstackclient
   

2. HashiCorp packer. The installation instructions still rely on the deprecated apt-key, so we provide the updated approach below.

   sudo mkdir -p /etc/apt/keyrings
   keyPath="/etc/apt/keyrings/hashicorp-archive-keyring.gpg"
   debArch="$(dpkg --print-architecture)"
   debUrl="https://apt.releases.hashicorp.com"
   debRel="$(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs)"
   curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor | \
     sudo tee $keyPath > /dev/null
   echo "deb [arch=$debArch signed-by=$keyPath] $debUrl $debRel main" | \
     sudo tee /etc/apt/sources.list.d/hashicorp.list
   sudo apt-get update && sudo apt-get install packer
   

In addition, you will need the openrc file for the project where you would like to store the images.

Process outline¶

• Create Packer template file
• Create Packer variable file
• Format and validate the template
• Build image. In particular, packer will do the following
  • Launch a VM using the specified specs and source image
  • Run customization commands
  • Shut down the VM
  • Take a snapshot and convert it into an image
  • Destroy the VM

First, we will see how to build an Ubuntu image. Then the same process will be shown for RockyLinux and AlmaLinux. Finally, we will provide some tips for debugging.

You may want to source your openrc file at this point as several steps will use the openstack client. Also, make sure that the python virtual environment is activated.

source myproj.uzh-openrc

Packer template¶

For Ubuntu 26 image, we can start with the following template. We will save it in the file named deb-based.pkr.hcl.

packer {
  required_plugins {
    openstack = {
      version = "~> 1"
      source  = "github.com/hashicorp/openstack"
    }
  }
}


variable "image_name" {
  type    = string
}

variable "source_image_url" {
  type    = string
}

variable "ssh_username" {
  type    = string
}

variable "network" {
  type    = string
}

variable "flavor" {
  type    = string
  default = "2cpu-8ram-hpcv3"
}

variable "image_visibility" {
  type    = string
  default = "private"
}


source "openstack" "deb" {
  external_source_image_url = var.source_image_url

  image_name       = var.image_name
  image_visibility = var.image_visibility
  flavor           = var.flavor
  ssh_username     = var.ssh_username
  networks         = [var.network]
  security_groups  = ["default"]
  ssh_wait_timeout = "20m"
  ssh_timeout      = "5m"
}


build {
  name    = "deb-compatible-image"
  sources = ["source.openstack.deb"]

  provisioner "shell" {
    inline = [
      "echo 'Starting provisioning...'",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get update -y",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y cloud-init python-is-python3"
    ]
  }

  # *** Place your provisioner(s) here ***

  provisioner "shell" {
    inline = [
      "echo 'Cleaning up...'",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get clean -y",
      "sudo cloud-init clean --logs --seed --machine-id",
      "sudo rm -rfv /home/*/.cache",
      "sudo rm -fv /home/*/.lesshst",
      "sudo rm -fv /etc/ssh/ssh_host_*",
      "sudo find /var/log -type f -exec truncate -s 0 {} \\;",
      "sudo rm -rfv /var/log/journal/*",
      "command -v fstrim && sudo fstrim -av || true",
      "sudo rm -fv /home/*/.bash_history",
      "sudo rm -fv /root/.ssh/authorized_keys",
      "sudo rm -fv /home/*/.ssh/authorized_keys"
    ]
  }

  post-processor "shell-local" {
    inline = [
      "echo 'Image build complete: ${var.image_name}'"
    ]
  }
}

The template defines what kind of VM packer should create, what source image to use, and what customization commands to execute. Custom commands are executed by provisioners. There are two of them defined. The first one updates the apt sources, runs upgrade just in case, ensures cloud-init is installed, and takes care of the python3 annoyance. The second provisioner cleans up the system to ensure that no instance-specific state or credentials are leaked into cloned systems. This prevents security risks such as duplicate host identities and unintended access, while ensuring each new instance generates its own unique configuration and keys on first boot.

You can place your provisioners between the two. Provisioners can transfer files to the image and they can execute various commands (see packer documentation for details).

As an example, let's create a provisioner that install some useful packages.

  provisioner "shell" {
    inline = [
      "echo 'Installing additional packages...'",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y language-pack-de",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y fd-find ripgrep fzf jq"
    ]
  }

This provisioner should be placed just below the comment # *** Place your provisioner(s) here ***.

Setting variables¶

There are four variables that are declared but not set in the template. In principle, you can set them directly in the template but they can also be defined in a separate file, so that you could reuse the template with different configurations.

The following parameters are required:

• image_name: name for the output image, e.g. "u26_20260421"
• source_image_url: source image url; it has to be a cloud image!
• ssh_username: SSH username, for Ubuntu it is ubuntu
• network: ID for the uzh-only network, e.g. openstack network show -f value -c ID uzh-only

We can save them into u26.pkrvars.hcl as follows.

cat <<EOF > u26.pkrvars.hcl
image_name       = "u26_20260518"
source_image_url = "https://cloud-images.ubuntu.com/resolute/20260421/resolute-server-cloudimg-amd64.img"
ssh_username     = "ubuntu"
network          = "$(openstack network show -f value -c ID uzh-only)"
EOF

Alternatively, you can set all these variables on command line as -var 'ssh_username=ubuntu' but the command becomes rather long in that case.

Initialize openstack plugin¶

Before running packer for the first time, you need to initialize the openstack plugin, which is defined at the top of the template. The following command will download and install the plugin. Afterwards, you do not need to run init unless you add more plugins to the template.

packer init deb-based.pkr.hcl

Format and validate template¶

It is recommended to keep the formatting consistent and to validate the template before building.

packer fmt deb-based.pkr.hcl
packer validate -var-file=u26.pkrvar.hcl deb-based.pkr.hcl

Building¶

Depending on the complexity of the provisioners, building may take anywhere from a few minutes to several hours.

packer build --var-file=u26.pkrvar.hcl deb-based.pkr.hcl

If the process fails, packer will remove the VM automatically.

Red Hat derivatives¶

Rocky Linux and Alma Linux are popular distributions that provide cloud images. Since they are very similar, we can use the same template.

packer {
  required_plugins {
    openstack = {
      version = "~> 1"
      source  = "github.com/hashicorp/openstack"
    }
  }
}


variable "image_name" {
  type = string
}

variable "source_image_url" {
  type = string
}

variable "ssh_username" {
  type = string
}

variable "network" {
  type = string
}

variable "flavor" {
  type    = string
  default = "2cpu-8ram-hpcv3"
}

variable "image_visibility" {
  type    = string
  default = "private"
}


source "openstack" "rh" {
  external_source_image_url = var.source_image_url

  image_name       = var.image_name
  image_visibility = var.image_visibility
  flavor           = var.flavor
  ssh_username     = var.ssh_username
  networks         = [var.network]
  security_groups  = ["default"]
  ssh_wait_timeout = "20m"
  ssh_timeout      = "5m"
}


build {
  name    = "rh-compatible-image"
  sources = ["source.openstack.rh"]

  provisioner "shell" {
    inline = [
      "echo 'Starting provisioning...'",
      "sudo dnf -y update",
      "sudo dnf -y install cloud-init langpacks-core-de",
    ]
  }

  # *** Place your provisioner(s) here ***

  provisioner "shell" {
    inline = [
      "echo 'Cleaning up...'",
      "sudo dnf -y autoremove",
      "sudo dnf clean all",
      "sudo cloud-init clean --logs --seed --machine-id",
      "sudo rm -rfv /home/*/.cache",
      "sudo rm -fv /home/*/.lesshst",
      "sudo rm -fv /etc/ssh/ssh_host_*",
      "sudo find /var/log -type f -exec truncate -s 0 {} \\;",
      "sudo rm -rfv /var/log/journal/*",
      "command -v fstrim >/dev/null && sudo fstrim -av || true",
      "sudo rm -fv /home/*/.bash_history",
      "sudo rm -fv /root/.ssh/authorized_keys",
      "sudo rm -fv /home/*/.ssh/authorized_keys"
    ]
  }

  post-processor "shell-local" {
    inline = [
      "echo 'Image build complete: ${var.image_name}'"
    ]
  }
}

As an example, we can create a provisioner that installs NVIDIA driver and CUDA toolkit. While the example is a good use case illustration, it is not appropriate for testing or exploration. The driver is large and may require 15-30 minutes to install.

  provisioner "shell" {
    environment_vars = [
      "urlPrefix=https://developer.download.nvidia.com/compute/cuda/repos",
      "distro=rhel10",
      "arch=x86_64"
    ]
    inline = [
      "echo 'Installing NVIDIA driver and CUDA toolit...'",
      "sudo dnf config-manager --add-repo $urlPrefix/$distro/$arch/cuda-$distro.repo",
      "sudo dnf install -y epel-release",
      "sudo dnf install -y cuda-driver cuda-toolkit"
    ]
  }

Rocky¶

Variable file

cat <<EOF > rocky.pkrvars.hcl
image_name       = "rocky_10.1"
source_image_url = "https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base-10.1-20251116.0.x86_64.qcow2"
ssh_username     = "rocky"
network          = "$(openstack network show -f value -c ID uzh-only)"
EOF

Building

packer validate -var-file=rocky.pkrvars.hcl rh-based.pkr.hcl
packer build -var-file=rocky.pkrvars.hcl rh-based.pkr.hcl

Alma¶

Variable file

cat <<EOF > alma.pkrvars.hcl
image_name       = "alma_10.1_20260518"
source_image_url = "https://mirror.init7.net/almalinux/10.1/cloud/x86_64/images/AlmaLinux-10-GenericCloud-10.1-20260518.0.x86_64.qcow2"
ssh_username     = "almalinux"
network          = "$(openstack network show -f value -c ID uzh-only)"
EOF

Building

packer validate -var-file=alma.pkrvars.hcl rh-based.pkr.hcl
packer build -var-file=alma.pkrvars.hcl rh-based.pkr.hcl

Debugging suggestions¶

Since the whole process may be rather time consuming, we suggest the following approach.

1. Create a simple image using one of the templates above without any additional provisioners.

2. Use the created image to instantiate a VM

3. Run your customization commands to ensure everything works correctly

4. Add a provisioner with those commands

5. Build a new image

To use an existing ScienceCloud image as a starting point, you would need to make the following modifications.

1. Replace source_image_url variable with source_image

   variable "source_image" {
     type    = string
   }
   

2. In the source block, replace external_source_image_url = var.source_image_url line with source_image = var.source_image

3. Remove the first provisioner as the update and upgrade already run and no longer needed.

4. Find the ID of the image, e.g. openstack image show -f value -c ID $imgName

5. In the variables file, replace source_image_url with source_image and set it to the image ID found in the previous step.

6. Build an image with the updated template and variables file.

If you still encounter problems, you can create an image with partial customization and use it as starting point. Packer will still need to instantiate VM but you will be able to avoid external image download, update, and upgrade calls that may be rather slow.