Quick start¶

The goal of this short tutorial is to introduce new users to ScienceCluster computing service. It assumes that the readers already have experience with remote Linux servers but not necessarily with clusters. For a longer tutorial, considering following the training handout or joining a training course.

Connecting to the cluster¶

You can connect to the cluster using your UZH shortname and Active Directory (AD) password, like so

ssh -l shortname cluster.s3it.uzh.ch

After running the command, you will be prompted for your password. Please note that there will be no echo as you type your password -- nothing will be displayed when you type each character. If you need to update your AD password, you can do so in the Identity Manager.

For more detailed instructions about how to connect with passwordless authentication, see here.

Data storage¶

There are three filesystems where you can store your data.

Your home filesystem (/home/$USER) has a quota of 400 GB. Typically, it is used to store configuration files, source code, software environments, and other important files.

Large input data and computational results can be temporarily stored on the scratch filesystem (~/scratch or /scratch/$USER), which has a quota of 20 TB. Please note that this filesystem is meant for temporary storage and the files may be automatically deleted if they have not been accessed within one month.

If you need additional space for persistent data beyond the home and scratch filesystems, you can use scalable storage. It is not subject to quota but it requires cost contributions based on the actual usage.

More information...

Job scheduling¶

Jobs are submitted with the sbatch command. The default values for resource allocations are very low. If you do not specify any parameters, Slurm (the automatic job allocation system) will allocate 1 vCPU, 1 MB of memory, and 1 second for execution time. Therefore, you need to specify at minimum the amount of memory and the expected runtime. For example, to run a hostname command on the cluster, you can create a file named test.job with the following contents:

#!/usr/bin/bash -l
hostname

Then you can submit it for execution with the following command.

sbatch --time=0:10:0 --mem=7800 --cpus-per-task=2 test.job

This will request 2 CPUs and 7800 MB of RAM for 10 minutes. Alternatively, you can specify these parameters in your job file; e.g.,

#!/usr/bin/bash -l
#SBATCH --time=0:10:0
#SBATCH --mem=7800
#SBATCH --cpus-per-task=2
hostname

You can use modules to request specific hardware types such as GPUs. All such modules are listed in the flovors section. For example, if you want to use H100 GPUs, you can load the h100 module, i.e.

module load h100

This will automatically limit the job submission to H100 nodes.

For testing or debugging purposes, you can run your job in an interactive session. The use of interactive sessions for computational work is generally discouraged. You can start an interactive session with the following command.

srun --pty --time=1:0:0 --mem-per-cpu=7800 --cpus-per-task=2 bash -l

An example with a GPU could be

module load t4
srun --pty --time=1:0:0 --mem-per-cpu=7800 --cpus-per-task=2 --gpus=1 bash -l

More information...

Maximum running time¶

You should strive to split your calculations into jobs that can finish in fewer than 24 hours. Short jobs are easier to schedule; i.e., they are likely to start earlier than long jobs. If something goes wrong, you might be able to detect it earlier. In case of a failure, you will be able to restart calculations from the last checkpoint rather than from the beginning. Finally, long jobs fill up the queue for extended periods and prevent other users from running their smaller jobs.

A job's runtime is controlled by the --time parameter. If your job runs beyond the specified time limit, Slurm will terminate it. Depending on the value of the --time parameter, slurm automatically places jobs into one of the quality of service (QOS) groups, which define the total number of jobs and the combined amount of resources a user can request. Longer QOS have lower limits. ScienceCluster has three different QOS groups.

• normal: 24 hours
• medium: 48 hours
• long: 7 days

More information...

Job management¶

You can view the list of currently scheduled and running jobs with the squeue command. Without any parameters, it will display all the jobs that are currently scheduled or running on the cluster. To see only your jobs, you need to specify the -u parameter.

squeue -u $USER

If you want to delete a job from the queue you can do so with scancel, and you need to specify the Job ID as an argument. The Job ID is always reported when you schedule a job. You can also find it in the output of squeue. Multiple jobs can be deleted at once. For example,

scancel 2850610 2850611

You can also cancel all your jobs at once without specifying and Job IDs. The following two commands delete all your jobs or all your pending jobs, respectively.

scancel
scancel --state=PENDING

More information...

Parallelisation¶

There are four main approaches to parallelisation.

• Single program that runs multiple processes each with private memory allocation
• Several program instances that run in parallel (i.e., job arrays)
• Single master program that launches several slave programs
• Single program that runs multiple processes with shared memory (MPI)

For the first approach, you do not need to do anything special. You just submit a job requesting the number of vCPUs that your program can efficiently use. The other three approaches are described in the Job Scheduling and Parallelisation sections of the documentation.

External resources¶

In addition to the documentation provided on this site, you can also find the following external resources useful.

• Slurm Quick Start Guide
• Slurm Documentation
• CECI Slurm Quick Start Tutorial (Although it is written specifically for CECI users, the tutorial is excellent and can be used as a general Slurm guide.)