Chimera: University HPC Cluster Tutorial

Material for training held on 15.10.2025

What is Chimera

When to use Chimera

  • It is a medium-sized cluster.
  • It is unsuitable for jobs requiring enormous resources and is not a substitute for supercomputers or the LHC Grid.
  • It is suitable for medium-sized tasks:
    • Up to a few thousand parallel processes
    • Up to a few Terabytes of data
    • Jobs that need fast turnaround, e.g.
      • final stages of data analysis
      • preparation and debugging of the SW
      • student projects

JupyterHub

  • The simplest access to the cluster is available via a web browser.
  • In your browser, go to the following URL:
    https://hpc.troja.mff.cuni.cz:8000
  • Note the “:8000” suffix. It must be there!
  • You will be asked for credentials. Put in your SIS login name and password:

  • Specify the parameters of your interactive job:
  • The JupyterHub launches the interactive job according to your specifications.
  • Once the job starts, you can use a terminal and jupyter notebooks from your browser window.
  • NOTE: If you are a new user, please log in to JupyterHub for your first login. It will add you to a default account “ffa”. Otherwise, you won’t be able to submit any jobs.

Logging in to the cluster using SSH

In the terminal window (on Mac or Linux) or PowerShell (on Windows), use the ssh command:

ssh <sis_login>@hpc.troja.mff.cuni.cz
  • Where <sis_login> is your user name in the university’s LDAP database (i.e the user name you use in SIS, CIS, and other university applications)
  • This will log you into the head node of the cluster.
  • Your home folder:
    /home/<sis_login>
  • Feel free to look around:
    • The command “ls“ lists the content of your home folder
    • The command  “df -h” shows you available storage devices
    • The command “squeue” will show you all running jobs on the cluster
  • Do not use the head node to run CPU-intensive processes.
    We will use jobs for this (see the following sections).

Chimera partitions

  • The cluster is subdivided into partitions, serving as queues for jobs with different priorities.
  • All users have access to the following partitions:
    • Free-for-all partitions: “ffa“, “ffa-preempt“, “ffa-check”
      • Low priority → it may take some time for your jobs to start. Depends on the cluster occupancy.
      • However, it offers access to the highest number of nodes – most cluster nodes are included in this partition.
      • The “preempt” and “check” partition jobs can be killed and resubmitted when fighting for resources with a higher priority job → your jobs must be able to recover from this if you want to use “ffa-preempt / ffa-check” partition.
      • Free-for-all partitions are limited for jobs running up to 1.5 days (ffa-preempt) and 12h (ffa).
    • Education partition: “edu”
      • Meant to be used for teaching
      • High-priority partition, but limited to only two cluster nodes (mff-a2-01 and mff-a2-01).
      • Limited to 4h jobs
      • Only one job is allowed per user.
    • GPU partition: “ffa-gpu”
      • For running GPU jobs (see later)
  • In addition, if you are a member of “ucjf” account, you gain access to the “ucjf” partition:
    • High priority
    • Unlimited time
    • Only five server nodes were bought by our department: ucjf-asus1, ucjf-asus2, ucjf-asusb1, ucjf-asusb2, ucjf-a4-01
    • To be added to the UCJF account, please contact 
      Daniel.Scheirich@matfyz.cuni.cz

IMPORTANT NOTE: You must be associated with an “account” to submit jobs. When you log in for the first time via JupyterHub (see above), you will be automatically added to the “ffa” account and will be able to use both “ffa” and “edu” partitions.

Submitting an interactive job

To submit and manage jobs (computing tasks), the cluster uses the SLURM batch system.

  • Interactive job: gives you terminal access to the worker node (behaves like ssh session). When you disconnect your terminal, the job is killed.
  • Batch job: your job is executed and runs until it is done. You cannot interact with your job.

Submitting an interactive job (use “ucjf” partition only if you have access to it):

salloc -p edu --mem 2G --cpus-per-task 2 -t 1:0:0 --reservation ucjf_174
salloc -p ucjf --mem 2G --cpus-per-task 2

It should give an output like this. Note that it may take some time for the job to start:

salloc: Granted job allocation 7725778
salloc: Waiting for resource configuration
salloc: Nodes mff-a2-02 are ready for job
sched3am@mff-a2-02:~$
  • NOTE: You can only have one job in the “edu” partition! If you already have a job running (e.g., because you have tried accessing it via JupyterHub), you must kill it before trying the command above. To kill your job:
      squeue --me
      scancel <ID_OF_THE_JOB_TO_KILL>
    The “ucjf” of “ffa” partitions do not have a job limit.
  • The interactive job gives you access to the worker node. You can run any CPU-intensive tasks there.
  • Option “-p edu” or “-p ucjf” specifies the partition where the job is executed. You must be a member of the UCJF account to be allowed to use the “ucjf” partition (see above).
  • Option “--mem 2G” specifies how much memory is allocated to your job. If the job runs out of memory, it is killed.
  • Option “--cpus-per-task 2” specifies how many CPUs should be given to your job. Note that hyperthreads are counted as CPU, so for most processors, one core equals two CPUs.
  • Option “-t 1:0:0” specifies run time to one hour. The Edu partition has a limit of 4h.
  • The options “--reservation ucjf_174” allow the job to use the reservation created for this training. Only available with the “edu” partition. Please do not use them outside of this training.

NOTE: Unlike SSH connections, interactive jobs do not provide a tunnel for displaying application windows. You need to use a workaround if you want to run GUI applications.

NOTE 2: An alternative way to submit interactive jobs:
srun -p edu --mem 2G --cpus-per-task 2 -t 1:0:0 --reservation ucjf_174 --pty bash -i
srun -p ucjf --mem 2G --cpus-per-task 2 --pty bash -i

Running GUI applications

  • Sometimes, it is helpful to run programs that produce windows
  • For example:
    • ROOT produces windows when displaying plots
    • Running Mathematica in GUI mode
  • Allowing applications to display windows must be done in two steps:
    1. Start the interactive job following the instructions in the previous section
    2. Open another terminal window on your laptop and connect to the head node with the “-Y” option. Once you are logged in, create another SSH connection to the server where your interactive job is running:
ssh <sis_login>@hpc.troja.mff.cuni.cz
ssh -Y <name_of_the_node_where_your_job_is_running>
  • SLURM will not allow you to SSH to the node where you have no running job. Therefore, your interactive job must run continuously while you work with the GUI application.
  • Your laptop OS must be capable of displaying X11 windows forwarded from the Linux OS. If you have Linux installed on your laptop, it will work automatically. You need third-party applications, such as XQuartz (Mac) or Xming (Windows), if you use a Windows or Mac operating system.

Example: running Mathematica in an interactive job

1st terminal window (only do this if you do not have the job running already):

ssh <sis_login>@hpc.troja.mff.cuni.cz
salloc -p edu --mem 2G --cpus-per-task 2 -t 1:0:0 --reservation ucjf_174

2nd terminal window:

ssh -Y <sis_login>@hpc.troja.mff.cuni.cz
ssh -Y <name_of_the_node_where_your_job_is_running>
module load Mathematica; mathematica

Example: running ROOT GUI in an interactive job

The two terminals were opened in the same way as before. In the 2nd terminal:

source /singularity/ucjf/root-6.28.12-x86_64-el9-gcc13-opt-LCG_104d_ATLAS_22/thisroot.sh 
root
TBrowser b

NOTE: There are several versions of ROOT installed on the cluster in the following location:

/singularity/ucjf/

You just need to initialise the version that you need for your work by running the “thisroot.sh” script from the appropriate folder.

Submitting batch jobs

  • Batch jobs run without requiring user interaction.
  • Job outputs are forwarded to the log files.
  • One can submit a large number of jobs using job arrays.
  • We must create an executable submit shell script to submit a batch job. Here is an example:
    • In your home folder, create a “tutorial_chimera” folder and change the directory: 
      mkdir tutorial_chimera
cd tutorial_chimera
    • Using your favourite editor (nano, vim, emacs), create a submit script: 
      nano submit.sh
#!/bin/bash
#SBATCH --array 0-9
#SBATCH --mem 500M
#SBATCH --cpus-per-task 2
#SBATCH --job-name=test
#SBATCH --error=test.%a.log
#SBATCH --output=test.%a.log
#SBATCH --open-mode append

# Here, do the real work
# Just a simple example:
python test.py $SLURM_ARRAY_TASK_ID
  • The slurm options are specified in the “#SBATCH” comment. When submitting the job, these options can also be set from the command line.
  • Option “--array” is used when running multiple jobs in parallel. The index of the sub-job is stored in the environment variable “$SLURM_ARRAY_TASK_ID” and “%a” placeholder.
  • Options “--error” and “--output” set the name(s) of the output log file(s). Use the “%a”  placeholder when submitting job arrays.
  • Option “--open-mode append” is useful when submitting to the “preempt” queues. When used, the log files are not overwritten by resubmitted jobs.
  • Now, create the “test.py” script. In our example, just a simple “Hello world” in Python:
import sys
print("Hello world", sys.argv[1] if len(sys.argv)>1 else "")

# wait for 20 seconds so that the job is not too fast
import time
for i in range(20):
  print("Working for", i, "seconds")
  time.sleep(1)
print("Done")
  • Finally, we need to make the shell script executable and submit it with the “sbatch” command:
chmod +x submit.sh
sbatch -p ffa-preempt submit.sh
  • Your job was now submitted to the “ffa-preempt” queue. It can take some time for the job to start, depending on the cluster occupancy.
  • You can monitor the job’s progress using the “squeue“ command: 
    squeue --me

Proper use of different queues

Cons and pros of different queues:

Partition pros cons
ucjf  – high priority
– unlimited run time
– usually short wait times		  – only five servers
– competing with your
colleagues for resources
ffa  – a large number of nodes
– less competition for resources		  – lower priority
– sometimes longer wait times
– runtime limited to 12h
ffa-preempt  – the entire cluster
– even less competition for resources		  – the same priority as ffa
– your jobs can be killed and
resubmitted → they must be
able to cope with this.
– runtime limited to 1.5 days

Partition decision tree:

Please be nice to your colleagues

  • Do not fill up the entire “ucjf” partition with long jobs
  • If you are submitting multiple parallel jobs, consider using job arrays to streamline your workflow. Job arrays can restrict the number of sub-jobs running in parallel:
#SBATCH --array 0-999%20
  • In this example, only 20 jobs from this array will run at once.

Checkpointing

Sometimes, you need to run a long job in a partition with a time limit. This can be done using a technique called checkpointing. When your job runs out of time (or is preempted), the complete state of your program will be saved to a checkpoint file and restored in another job.

Instructions on how to set up checkpointing for your job are given here:

https://gitlab.mff.cuni.cz/mff/hpc/clusters/-/tree/master/user_checkpointing_demo/checkpoint-restart-scripts?ref_type=heads

Storage

  • The command “df -h” shows you available storage devices and their occupancy.
  • /home: your home folder.
    • You put your code, work, logs, and small data files here (~ a few GB).
    • Do not put large data here. Like seriously, don’t!
  • /work: large storage with magnetic HDDs.
    • This is where you should put your large data.
    • The storage is connected via a fast network (InfiniBand), but HDDs have limited parallel access capability.
  • /archive: long-term storage.
    • Place the data you do not want to delete yet, but do not use on a daily basis, here.
    • Connected only by an Ethernet network (slower than IB)
  • /scratch: for smaller data (from 100s of GB units of TB)
    • Connected via InfiniBand
    • Based on solid-state drives (SSD). They are much better at handling parallel access.
  • /singularity: small disk we use for storing SW containers and virtual environments—writable from head-node, read-only from worker nodes.

Where to put your data

  • /work/tmp: accessible to everyone, but content can be deleted anytime without warning
  • /work/<your_user_name>: accessible to you. If you do not have this folder, contact the cluster administrator.
  • /work/ucjf-atlas: accessible for everyone in the hpc-atlas group.
    • If you want to have your own group and your shared folder on the “/work” storage, don’t hesitate to contact the cluster admin and provide a list of users to be included in the group.
  • /scratch/tmp: the same as “/work/tmp“
  • /scratch/ucjf-atlas: the same as “/work/ucjf-atlas“.
  • There are no automatic user folders on “/scratch“. You have to request to have a folder (or a group folder) on /scratch

Optimising storage parallel access

  • There is no absolute rule. You should test how your jobs are doing when reading data from many jobs in parallel.
    • You submit, e.g. 10 jobs that all read your data: 
      sbatch -p ucjf your_code.sh
    • Remember the job ID
    • Once jobs are over, run the “seff” command to measure the CPU efficiency of your job. 
      seff <the_finished_job_id>
      • If jobs are running at more than 70% CPU, you are probably okay. You can try increasing the number of parallel sub-jobs or keeping it the same.
      • If jobs are using <<50%, your jobs spend most of the time waiting in I/O sleep. You should reduce the number of parallel sub-jobs.
  • Job efficiency can also depend on the actions of other users.
  • Please remember you are not the only user of the cluster! Even if you feel your job is okay, you might actually slow down the cluster for other users.
  • General advice: It’s usually worth shrinking your data by creating reduced derived datasets (e.g., filtering events that do not meet the selection criteria, removing unused variables, etc.).

Containers and virtual environments

  • The cluster is (currently) running Rocky Linux 9 (EL9)
  • You can use containers if you need a different OS or additional software not installed on the cluster.
  • Chimera utilises the “Apptainer” (formerly Singularity) containers. They have an extension “.sif”
  • Some ~useful containers are already downloaded in the “/singularity/ucjf” folder (mostly thanks to Pavel Reznicek).
    • cc7.sif: CERN CentOS Linux 7. (Legacy Linux used at CERN)

    • ubuntu_v22.04.5_roots.sif: ubuntu + ROOT (note that ROOT is also directly installed on the cluster, so you do not need to use containers)

  • To activate the container, run the following command (first, run the interactive job):
salloc -p edu --mem 2G --cpus-per-task 2 -t 1:0:0 --reservation ucjf_174
apptainer exec --bind=/home --bind=/work --bind=/scratch --bind /singularity/ucjf:/singularity_ucjf /singularity/ucjf/cc7.sif /bin/bash
  • If pre-installed containers do not suit your needs, you can obtain a new container from Docker Hub.
    For example: (may take some time to download)
# We need more memory!
salloc -p ucjf --mem 50G --cpus-per-task 2
export APPTAINER_CACHEDIR=/scratch/tmp/<user_id>/apptainer_tmp
export APPTAINER_TMPDIR=/scratch/tmp/<user_id>/apptainer_tmp
apptainer pull docker://pytorch/pytorch
apptainer exec --bind=/home --bind=/work --bind=/scratch --bind /singularity/ucjf:/singularity_ucjf pytorch_latest.sif /bin/bash
  • When using containers in batch mode, you must execute your program within the container.
apptainer exec --bind=/home --bind=/work --bind=/scratch --bind /singularity/ucjf:/singularity_ucjf pytorch_latest.sif <script_to_execute>

NOTE: You cannot install new software into the container unless you have special privileges.

  • If you need extra SW in your base containers, you can ask Pavel Reznicek for help (he has the special privileges)
  • You can create the .sif container on your computer (laptop, office desktop, …) where you have superuser privileges and then copy the container file into the cluster.

Python environments

  • Bare Python is available on the cluster, but it does not have advanced libraries installed (e.g., Pandas, TensorFlow, Sympy, etc).
  • You can use Python virtual environments to configure your Python:
python -m venv <path_where_venv_is_store>

# Usually, the folder is named "venv":
python -m venv venv

# activate the environment
source venv/bin/activate

# install your libraries using pip
pip install pandas
  • NOTE: “venv” folders cannot be copied (for some reason, all paths inside are absolute). So, if you need to share a Python environment with your colleagues, you can put it in the ” /singularity/ucjf” folder
  • For example:
    • “/singularity/ucjf/venv_4top” used by the 4-top analysis team
    • “/singularity/ucjf/venv_htt” used by the 4-top H->tautau analysis
    • “/singularity/ucjf/venv_tf_218” used in the ML class
  • Activating the “shared” venv: 
    source /singularity/ucjf/venv_tf_218/bin/activate

Integrating Visual Studio Code with the cluster

  • Run the Visual Studio Code app (called just “code” on Linux)
  • In the left panel, click “Extensions” and install the “Remote-SSH” extension.
  • In the bottom left corner, click on the icon:
  • The menu will pop up. Choose “connect a current window to host” and then put in the cluster URL: 
    ssh <your_user_name>@hpc.troja.mff.cuni.cz
  • Now, you can work with the files on the cluster as if they were stored locally.
  • WARNING: The Visual Studio Code runs a server application on the cluster’s head node.  Do not execute Python code (or Jupyter notebooks) directly from Visual Studio, as they will run on the head node.

Connecting Visual Studio Code to JupyterHub

  • Start the JupyterHub interactive job (see the JupyterHub section)
  • Go to the menu “File” → “Hub Control Panel” → “Token”
  • Click on “Request new API token” and copy the generated token string (Ctrl+C).
  • Connect the Visual Studio Code to the head node (as described in the previous section)
  • Install the JupyterHub extension
  • Create a new Jupyter Notebook file (e.g. “test.ipynb“)
  • In the top-right corner, click on the “Select Kernel” button
  • Choose “Existing Jupyter Hub server…” and “Enter the URL of he running JupyterHub Server…”
  • Add URL: https://hpc.troja.mff.cuni.cz:8000
  • Enter your username
  • Enter the copied token (Ctrl+V). Do not use a password; it will not work.
  • Name the session (e.g. JupyterHub1)
  • Choose the Python Kernel (e.g. “Python with tf2.18 and more” or whatever you need)
  • Now, the content of your notebook will be executed on the worker node allocated by JupyterHub rather than by the head node.

NOTE: In case you have problems connecting, you can try to check the following option in the “settings” menu of Visual Studio Code:

 

GPU

  • Two NVIDIA L40 GPUs are available on Chimera (more will come next year).
  • You must use a special partition and the slurm option to get access to the GPU:
salloc -p gpu-ffa --gres "mps:5" --mem 10G -t 12:0:0 --cpus-per-task 2
  • Option “-p gpu-ffa” specifies to use of the partition with the GPU
  • Option “--gres 'mps:5'” specifies you want to use 5% of the GPU capacity
    • To use the full GPU (typical use case), specify “--gres 'gpu:1'“
    • Sharing a GPU among multiple users is practical for teaching
  • To utilise the GPU, a special library (CUDA) is required, which is not included in the base system. You need to use the container to get the libraries.
  • There is a pre-installed “venv” in the  “/singularity” folder
source /singularity/JupyterHUB/tf/venv/bin/activate
  • Now, you can check if the GPU is visible. Execute “python” and copy the following code into the terminal:
import tensorflow as tf 
print("Available GPUs:", tf.config.list_physical_devices('GPU'))
  • You should see if the GPU is available
  • You can also create your own venv with additional libraries if need be. Note that you need to specify to install CUDA libraries alongside the TensorFlow or Torch frameworks: 
    python -m venv my_venv
source my_venv/bin/activate

# to install TensorFlow
pip install 'tensorflow[and-cuda]'

# to install torch
pip install torch

GPU and Jupyter Notebooks

  • C.f. section Connecting Visual Studio Code to JupyterHub
  • Choose “Python with tf 2.18 and more” to gain access to TensorFlow libraries and a GPU.
  • NOTE: to be able to use the GPU, you need to run the job with “-p ffa-gpu --gres "gpu:1"” options, as described above.

Creating your own Jupyter Kernel

  • However, if you want to add your own libraries, you may want to create your own kernel:
mkdir -p ~/.local/share/jupyter/kernels/my_kernel
cd ~/.local/share/jupyter/kernels/my_kernel
  • Create a config file “kernel.json“: 
    {
  "argv": [
    "/home/<your_user_name>/.local/share/jupyter/kernels/my_kernel/start.sh",
    "{connection_file}"
  ],
  "display_name": "my_kernel",
  "language": "python",
  "metadata": {
    "debugger": true
  }
}
  • Create a “start.sh” script: 
    #!/bin/bash
source /home/<your_user_name>/.local/share/jupyter/kernels/my_kernel/venv/bin/activate
python -m ipykernel_launcher -f $@
  • Make the “start.sh” script executable: 
    chmod +x start.sh
  • Finally, you have to create your own virtual environment (see above) where you install your Python libraries. It is important to install the package ipykernel into the venv; otherwise, the kernel will not work: 
    python -m venv venv
source venv/bin/activate
pip install ipykernel
pip install <your other libraries>
  • If you have done everything correctly, your notebooks (both in JupyterHub and Visual Studio Code) should see your new kernel “my_kernel“.

Creating your own PyROOT Kernel

  • If you want to use PyROOT in Jupytrer notebooks, you can create the following kernel:
mkdir -p ~/.local/share/jupyter/kernels/root_kernel
cd ~/.local/share/jupyter/kernels/root_kernel
  • Create a config file “kernel.json“: 
    {
  "argv": [
    "/home/<your_user_name>/.local/share/jupyter/kernels/root_kernel/start.sh",
    "{connection_file}"
  ],
  "display_name": "root_kernel",
  "language": "python",
  "metadata": {
    "debugger": true
  }
}
  • Create a “start.sh” script: 
    #!/bin/bash
source /singularity/ucjf/root-6.28.04-x86_64-el9-gcc11-opt-LCG104/root-config.sh 
python -m ipykernel_launcher -f $@
  • Make the “start.sh” script executable:
    chmod +x start.sh

If you have done everything correctly, your notebooks (both in JupyterHub and Visual Studio Code) should see your new kernel “root_kernel“. You should now be able to import “ROOT” into your notebooks

from ROOT import TCanvas, TF1
%jsroot on
f = TF1("sin", "sin(x)", 0, 10)
c = TCanvas()
f.Draw()
c.Draw()
  • NOTE: With the recent versions of ROOT, I was unable to combine ROOT and venv environments in a single kernel.