Containers (Singularity) Tutorial¶

This tutorial demonstrates how to use Singularity to create a container based software environment for use on ScienceCluster (or elsewhere).

Workflow Overview¶

Before beginning, it's helpful to understand the basic parts of a Singularity workflow.

First, you need to create a container. You can either obtain a container from a container registry such as docker hub or build it from your own definition file. In the first case, it should be possible to create the container directly on ScienceCluster. However, if you need some customizations such as installing custom applications, combining multiple frameworks, or adding custom configuration, you may need to build a sif image elsewhere (e.g. on a ScienceCloud VM) and transfer the image to ScienceCluster.

Once your container is ready, you would need to prepare your code to run from the Singularity environment. This step will involve augmenting your Slurm submission script to use the Singularity container.

Building from container registry¶

Container registries such as docker hub offer a wide range of prebuilt containers. They do not need to be built on ScienceCluster but rather converted to the appropriate singularity format. This could typically be done directly on ScienceCluster. Popular containers available through docker hub include:

• PyTorch
• Tensorflow
• Rocker Project
  • r-base
  • tidyverse
  • geospatial

Once you've found the software that you want, copy the relevant container ID from the provided docker pull ... command. You should prepend docker:// to this ID and use it in the singularity build command as shown below. Using docker://rocker/tidyverse will pull the latest available version of the software installed in the container. You can view various available versions of tidyverse via the Tags subpage. For example, if you specifically need R 4.3.1, you can copy the ID from the corresponding docker pull ... command. Thus, the full string to use in the singularity build command will be docker://rocker/tidyverse:4.3.1.

For example, you can create a rocker container with R and tidyverse packages as follows. We recommend to use an interactive session as building often requires more memory than available to users on the login nodes.

srun --pty -n 1 -c 2 --time=00:30:00 --mem=7G bash -l
module load singularityce
singularity build /data/$USER/tidyverse.sif docker://rocker/tidyverse

The first command opens an interactive session. The second loads the singularity module. The final command builds the container in the /data/$USER/tidyverse.sif file. Please note that the container file tidyverse.sif will be in your data directory. If you save containers in your home directory, you will exceed your quota very quickly.

If you need a specific version (tag) rather than the latest one, you can add it after the column.

singularity build /data/$USER/tidyverse.sif docker://rocker/tidyverse:4.3.1

If the building process fails due to insufficient permissions, you may have to build the container elsewhere, e.g. on a ScienceCloud VM, and transfer it to ScienceCluster.

Building a custom image¶

Creating a custom image from a definition file¶

If you need to add software to a pre-existing Singularity Image file, you'll need to use a custom definition file that "bootstraps" a Singularity or a Docker image. "Bootstrapping" an image means that you will be using the image as a starting point for building a custom image with additional software or configuration. Some building commands may require superuser (sudo) privileges. In that case, you would need to build the container either on a ScienceCloud VM (using a source VM image that has Singularity preinstalled; e.g., ***Singularity on Ubuntu 20.04 (2024-05-29)) or on your own computer. The installation directions can be found here. After the *.sif file has been created, you would need to transfer it to ScienceCluster.

Once you have Singularity available via a ScienceCloud VM or via your own laptop, you'll need to make a definition file. The definition file is a plain text file that selects the starting image that you'll bootstrap as well as additional commands that will add more software to the container.

An example Singularity definition file might look like the following:

Bootstrap: docker
From: tensorflow/tensorflow

%post
    pip install pandas

This example uses a TensorFlow container from DockerHub and includes a %post section to install the pandas package. In general, %post section allows users to define specific commands that can augment the container. The pip program is available as it was already installed in the Tensorflow container.

To find out what software is available in a container, you can either research the existing Docker Hub information on the container or use the singularity shell command to explore the container interactively.

A more complex example of a Singularity definition file might look like the following:

Bootstrap: docker
From: rocker/tidyverse:4.0.3

%post
    apt-get update && . /etc/environment
    wget sourceforge.net/projects/mcmc-jags/files/JAGS/4.x/Source/JAGS-4.3.0.tar.gz -O jags.tar.gz
    tar -xf jags.tar.gz
    cd JAGS* && ./configure && make -j4 && make install
    cd ~
    apt-get update && . /etc/environment
    wget sourceforge.net/projects/jags-wiener/files/JAGS-WIENER-MODULE-1.1.tar.gz -O jagswiener.tar.gz
    tar -xf jagswiener.tar.gz
    cd JAGS-WIENER-MODULE-1.1 && ./configure && make -j4 && make install
    R -e "install.packages('runjags')"

Notice that this example uses many operating system commands to prepare/install system level packages; for example, apt-get update and make install. You can use these commands because the rocker/tidyverse:4.0.3 container is built using Ubuntu 20 as the operating system. To determine this, you can pull the prebuilt image file from Docker hub using singularity pull docker://rocker/tidyverse:4.0.3 then use singularity shell tidyverse_4.0.3.sif to open a Command Line directly within the Container. When in the command line, you can run lsb_release -a to find the operating system version.

Once you've saved your definition file as a text file (e.g., using the file name recipe.def), you can then try to build a Singularity Image File from it using:

sudo singularity build tensorflow.sif recipe.def

In this example, the outputted *.sif file will be named tensorflow.sif. However, you can edit this name to be whatever you'd like.

Once you've created a Singularity Image File, you should transfer it to ScienceCluster.

Prepare your code to run from the Singularity environment¶

The final step to a Singularity workflow is preparing your code to use the environment. This step involves editing the Slurm submission script. Take for example the following submission script (more examples on this documentation page):

#!/usr/bin/bash -l
#SBATCH --time=00:10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=3000
#SBATCH --gpus=1
module load anaconda3
source activate tf
python examplecode.py

This code assumes that the tf environment has already been created, as it uses the source activate tf line to activate that environment. Then, it runs the examplecode.py script using python. To change this submission script workflow to use a Singularity container environment, consider the following submission script:

#!/usr/bin/bash -l
#SBATCH --time=00:10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=3000
#SBATCH --gpus=1
module load singularityce
singularity exec --nv tensorflow.sif python examplecode.py

The principal edits to this script are:

• source activate tf has been replaced by module load singularityce, which shows that Anaconda is no longer being used as the environment manager; Singularity will instead be used.
• Instead of simply using python examplecode.py to run the script of interest, there is now an extended singularity exec command. This command executes an arbitrary command of interest using the specified Singularity container environment. The flags to this command are crucial to understand:
  • The --nv flag allows the container to access NVIDIA drivers so that the container can take advantage of available GPU's (see here).
  • The tensorflow.sif argument specifies the container file created previously

Once you've augmented your Slurm Submission script, your code is ready to be submitted using a standard submission workflow (i.e., via sbatch after loading modules of interest).

Using an interactive session to explore a Singularity Container¶

As mentioned previously, it's often helpful to explore a Singularity container's environment interactively rather than via a submission script—especially when creating the Singularity Image file. To do so, first request and receive an appropriate interactive session on a ScienceCluster node, which will ensure you don't use too many resources on a login node.

Once you've received an interactive session that meets your computational needs, and after you've loaded the required modules (e.g., module load gpu), you can then request a Singularity shell prompt from the Command Line that will open inside of the container environment. For example:

singularity shell --nv tensorflow.sif

will open a shell in the tensorflow.sif Singularity container.

Singularity Cache Cleaning¶

If you commonly build Singularity Image Files on ScienceCluster, you may notice that your /home folder is filling up with data. This is likely due to the Singularity cache being filled with reference files from the building processes. To clean the cache, simply run:

singularity cache clean