Skip to content
BranchesTags

Latest commit

 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
README.md
README.md
 
 

Repository files navigation

Building a Docker Container on Alps

This guide explains how to build and run a Docker container on the Alps system using Podman, Enroot, and SLURM. Follow the steps carefully to set up your environment and run workloads inside a container.

STEP-1: Create a Project Folder

Navigate to your project location and create a folder to store your Dockerfile and the .sqsh file (SquashFS image) generated from the Docker image.

cd /capstor/store/cscs/sdsc/<project-name>
mkdir MyDocker && cd MyDocker

STEP-2: Create a Dockerfile

FROM nvcr.io/nvidia/pytorch:24.01-py3
ENV DEBIAN_FRONTEND=noninteractive
# Install python venv and OpenCV dependencies, then clean up
RUN apt-get update && apt-get install -y \
    python3.10-venv \
    libgl1 \
    libglib2.0-0 \
    libsm6 \
    libxrender1 \
    libxext6

STEP-3: Configure Podman Storage

Now that we’ve set up the Dockerfile, we can pass it to Podman to build a container. Podman requires some storage configuration. Create the following file:

vim $HOME/.config/containers/storage.conf

Insert this content:

[storage]
  driver = "overlay"
  runroot = "/dev/shm/$USER/runroot"
  graphroot = "/dev/shm/$USER/root"
[storage.options.overlay]
  mount_program = "/usr/bin/fuse-overlayfs-1.13"

STEP-4: Build and Convert the Container

To build a container with Podman, request a compute node shell from SLURM, pass the Dockerfile to Podman, and finally import the built container using Enroot.

STEP-4.1: Request an Interactive Session

srun --partition debug --pty bash

STEP-4.2: Build the Dockerfile

podman build -t ExampleDocker .

STEP-4.3: Convert to SquashFS

Use Enroot to convert the Podman image into an Enroot-compatible SquashFS image:

enroot import -x mount -o ExampleDocker.sqsh podman://ExampleDocker

Exit the SLURM allocation:

exit

At this point, you should see both the Dockerfile and the new .sqsh file, an example below:

ls -l /capstor/store/cscs/sdsc/<project-name>/MyDocker
...
-rw-r-----+ 1 ssaha sd24 16440033280 Aug  5 18:28 ExampleDocker.sqsh
-rw-rw----+ 1 ssaha sd24        1446 Aug  5 18:25 Dockerfile
...

STEP-5: Create an Environment Definition File (EDF)

We need to set up an EDF (Environment Definition File) to tell SLURM which container to use. Create the file:

vim ~/.edf/ExampleDocker.toml

Insert the following:

image = "/capstor/store/cscs/sdsc/<project-name>/MyDocker/ExampleDocker.sqsh"

mounts = ["/capstor", "/users"]

writable = true

[annotations]
com.hooks.aws_ofi_nccl.enabled = "true"
com.hooks.aws_ofi_nccl.variant = "cuda12"

[env]
FI_CXI_DISABLE_HOST_REGISTER = "1"
FI_MR_CACHE_MONITOR = "userfaultfd"
NCCL_DEBUG = "INFO"

STEP-6: Set Up Python Virtual Environment

STEP-6.1: Request Interactive Session with EDF

srun --environment=ExampleDocker --container-workdir=$PWD --pty bash

STEP-6.2: Create Python Virtual Environment

python -m venv --system-site-packages ./<ExampleEnv>

STEP-6.3: Activate Environment and Install Libraries

source /capstor/store/cscs/sdsc/<project-name>/ExampleDocker/ExampleEnv/bin/activate

Install required packages:

python -m pip install \
    omegaconf==2.3.0 \
    opencv-python==4.9.0.80 \
    tensorboardX==2.6 \
    tensorboard==2.15.1 \
    h5py==3.14.0 \
    scikit-image==0.25.2

Below is an example SLURM job script that uses the Docker environment (ExampleDocker.toml).

#!/bin/bash
#SBATCH --job-name=cad
#SBATCH --partition debug
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --exclusive
#SBATCH --time=00:15:00
##SBATCH --environment=taming-transformer-pytorch-250325
#SBATCH --account=sd24
#SBATCH --output=/capstor/scratch/cscs/ssaha/Experiments/CAD-August-8-2025/daint_exp_07-08-2025-001_debug/bs_0_ngpu_4_250808_1141_70540/slurm_out_%j.out  # Output log file

CONFIG=no-defined
CODE_PATH=/capstor/scratch/cscs/ssaha/Jobs/cad
CODE_TARGZ_FILE_NAME=250808_114151_cd6941f6.tar.gz

echo "*** CONFIG ***"
echo $CONFIG
echo "*** CODE_PATH ***"
echo $CODE_PATH
echo "*** CODE_FOLDER_NAME ***"
echo 250808_114151_cd6941f6
echo "*** CODE_TARGZ_FILE_NAME ***"
echo $CODE_TARGZ_FILE_NAME

# Path to your Docker environment config
TOML_PATH="/users/ssaha/.edf/ExampleDocker.toml"

# Distributed training setup
# export MASTER_ADDR=$(hostname)  # Activate if you use DistributedDataParallel instead of DataParallel in PyTorch
# export MASTER_PORT=29501        # Activate if you use DistributedDataParallel instead of DataParallel in PyTorch
export OMP_NUM_THREADS=4

# Setting the codebase path to PYTHONPATH
export PYTHONPATH=/capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad/:$PYTHONPATH
echo "*** PYTHONPATH ***"
echo $PYTHONPATH

srun --export=ALL,CONFIG="$CONFIG",CODE_PATH="$CODE_PATH",CODE_TARGZ_FILE_NAME="$CODE_TARGZ_FILE_NAME" \
     --environment=$TOML_PATH \
     -u -l \
     bash -c '

  # Extract codebase
  echo "mkdir -p /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad"
  mkdir -p /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad

  echo "tar -xzf $CODE_PATH/$CODE_TARGZ_FILE_NAME -C /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad/ --strip-components=1"
  tar -xzf $CODE_PATH/$CODE_TARGZ_FILE_NAME -C /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad/ --strip-components=1

  # Change directory to code folder
  echo "cd /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad/"
  cd /capstor/scratch/cscs/ssaha/Code/250808_114151_cd6941f6/cad/

  echo "[Node $SLURM_PROCID] In container: $(hostname)"
  echo "[Node $SLURM_PROCID] PWD: $(pwd)"

  # ✅ Activate your virtual environment inside the container
  echo "source /capstor/store/cscs/sdsc/sd24/docker4cad/python_venv/cad/bin/activate"
  source /capstor/store/cscs/sdsc/sd24/docker4cad/python_venv/cad/bin/activate

  set -x

  echo ""
  echo "*** which python"
  which python

  # echo "*** which torchrun"
  # which torchrun

  # ✅ Optional sanity checks
  python -c "import torch; print(\"Torch version:\", torch.__version__)"
  python -c "import omegaconf; print(\"OmegaConf OK\")"
  python -c "import cv2; print(\"cv2 OK\")"

  # Numeric safety settings
  export NVIDIA_TF32_OVERRIDE=0                  # Disable TF32 in all CUDA libraries
  export TORCH_FLOAT32_MATMUL_PRECISION=highest  # Use highest precision for PyTorch GEMMs
  echo "*** Numeric Safety Settings ***"
  echo "NVIDIA_TF32_OVERRIDE=$NVIDIA_TF32_OVERRIDE"
  echo "TORCH_FLOAT32_MATMUL_PRECISION=$TORCH_FLOAT32_MATMUL_PRECISION"

  # Use this for Distributed Data Parallel
  # python -m torch.distributed.run --standalone --nproc-per-node=4 main.py $CONFIG

  # Use this for Data Parallel
  export CUDA_VISIBLE_DEVICES=0,1,2,3
  python -m scripts_2_5d_3d/main_CAD.py
'

echo "✅ Finished at: $(date)"

Running more than one job step per node

Running multiple job steps in parallel on the same allocated set of nodes can improve resource utilization by taking advantage of all the available CPUs, GPUs, or memory within a single job allocation. Please visit the CSCS documentation for more details. Below is an example SLURM job script that submits four training jobs to the four GPUs on a single compute node.

#!/bin/bash
## SBATCH --partition=debug # TODO
#SBATCH --job-name=cad
#SBATCH --exclusive
#SBATCH --mem=450G
#SBATCH -N1
#SBATCH --time=24:00:00
#SBATCH --account=sd24
#SBATCH --output=/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/SLURM_OUTPUT_FILES/251014_2117_93f05/slurm_master_%j.out


TOML_PATH="/users/ssaha/.edf/docker4cad.toml"

echo "*** code_path ***"; echo "/capstor/scratch/cscs/ssaha/Jobs/cad"
echo "*** CODE_FOLDER_NAME ***"; echo "251014_211708_6cc5eae4"
echo "*** code_targz_file_name ***"; echo "251014_211708_6cc5eae4.tar.gz"

# ---------- 1) Prep (no GPU) ----------
echo ">>> [prep] Extracting code and verifying environment..."
mkdir -p /capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad
tar -xzf "/capstor/scratch/cscs/ssaha/Jobs/cad/251014_211708_6cc5eae4.tar.gz" -C /capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad --strip-components=1



# ---------- 2) Parallel training helper (1 GPU per step) ----------
launch_step () {

  local SET="$1"; shift
  local LOG="/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/SLURM_OUTPUT_FILES/251014_2117_93f05/slurm_${SET}_%j.out"
  local ARGS=("$@")
  mkdir -p -- "$(dirname "$LOG")"

  srun --export=NONE \
  --environment="$TOML_PATH" \
  -N1 \
  --ntasks-per-node=1 \
  --exclusive \
  --gpus-per-task=1 \
  --cpus-per-gpu=16 \
  --mem=50G \
   -u -l \
   --output="$LOG" \
   bash -c '
    set -euo pipefail

    echo "[Node $SLURM_PROCID] In container: $(hostname)"
    echo "[Node $SLURM_PROCID] PWD: $(pwd)"

    source /capstor/store/cscs/sdsc/sd24/docker4cad/python_venv/cad/bin/activate

     set -x

    echo \" \"
    echo \"*** which python\"
    which python

    echo \"*** which torchrun\"
    which torchrun

    # ✅ Optionally sanity check
    python -c "import torch; print(\"*** Torch version: ***\", torch.__version__)"
    python -c "import omegaconf; print(\"*** OmegaConf OK ***\")"
    python -c "import cv2; print(\"*** cv2 OK ***\")"

    # step-local thread
    export OMP_NUM_THREADS=1
    export MKL_NUM_THREADS=1
    export OPENBLAS_NUM_THREADS=1

    cd "/capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad"
    echo "PWD: $(pwd)"
    export PYTHONPATH="/capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad:/capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad/scripts_2_5d_3d:${PYTHONPATH:-}"

    mkdir -p -- "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/caches" "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/models"

    PYMAIN="/capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad/scripts_2_5d_3d/main_CAD_modified_October_10_2025.py"
    [ -f "$PYMAIN" ] || { echo "FATAL: $PYMAIN not found" >&2; ls -la "/capstor/scratch/cscs/ssaha/Code/251014_211708_6cc5eae4/cad"; exit 2; }

    python "$PYMAIN" \
    "$@"
    ' _ "${ARGS[@]}" &
}

# ---------- 3) Four parallel steps ----------
launch_step "SET-1" \
  --cfg "2d_cremiA_bs16_ps256_loss0.0_slice1.0_cross1.0_interaction1.0" \
  --cfg_3d "3d_cremiA_h160_noft_lr_ratio1_ft10000" \
  --cache_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/caches" \
  --save_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/models" \
  --resume "false" \
  --model_id -1 \
  --model_name "" \
  --display_freq 100 \
  --valid_freq 10000 \
  --save_freq 10000 \
  --batch_size 16 \
  --num_workers 4 \
  --batch_size_3d 2 \
  --num_workers_3d 2 \
  --data_folder "/capstor/scratch/cscs/ssaha/Datasets/PSI/ALBUM" \
  --dataset_name "cremiA" \
  --valid_dataset "cremiA" \
  --dataset_name_3d "cremiA"

launch_step "SET-2" \
  --cfg "2d_cremiA_bs16_ps256_loss0.0_slice1.0_cross1.0_interaction1.0" \
  --cfg_3d "3d_cremiA_h160_noft_lr_ratio1_ft10000" \
  --cache_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/caches" \
  --save_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/models" \
  --resume "false" \
  --model_id -1 \
  --model_name "" \
  --display_freq 100 \
  --valid_freq 10000 \
  --save_freq 10000 \
  --batch_size 16 \
  --num_workers 4 \
  --batch_size_3d 2 \
  --num_workers_3d 2 \
  --data_folder "/capstor/scratch/cscs/ssaha/Datasets/PSI/ALBUM" \
  --dataset_name "cremiB" \
  --valid_dataset "cremiB" \
  --dataset_name_3d "cremiB"

launch_step "SET-3" \
  --cfg "2d_cremiA_bs16_ps256_loss0.0_slice1.0_cross1.0_interaction1.0" \
  --cfg_3d "3d_cremiA_h160_noft_lr_ratio1_ft10000" \
  --cache_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/caches" \
  --save_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/models" \
  --resume "false" \
  --model_id -1 \
  --model_name "" \
  --display_freq 100 \
  --valid_freq 10000 \
  --save_freq 10000 \
  --batch_size 16 \
  --num_workers 4 \
  --batch_size_3d 2 \
  --num_workers_3d 2 \
  --data_folder "/capstor/scratch/cscs/ssaha/Datasets/PSI/ALBUM" \
  --dataset_name "cremiC" \
  --valid_dataset "cremiC" \
  --dataset_name_3d "cremiC"


launch_step "SET-4" \
  --cfg "2d_wafer4_bs16_ps256_loss0.0_slice1.0_cross1.0_interaction1.0_original" \
  --cfg_3d "3d_wafer4_h160_noft_lr_ratio1_ft10000_original" \
  --cache_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/caches" \
  --save_path "/capstor/scratch/cscs/ssaha/Experiments/ALBUM-CAD-October-14-2025-001/models" \
  --resume "false" \
  --model_id -1 \
  --model_name "" \
  --display_freq 100 \
  --valid_freq 10000 \
  --save_freq 10000 \
  --batch_size 16 \
  --num_workers 4 \
  --batch_size_3d 2 \
  --num_workers_3d 2 \
  --data_folder "/capstor/scratch/cscs/ssaha/Datasets/PSI/ALBUM" \
  --dataset_name "wafer4" \
  --valid_dataset "wafer4" \
  --dataset_name_3d "wafer4"

wait
echo "✅ All jobs finished at: $(date)"


### 📚 BibTeX Citation  

```bibtex
@misc{cscs_llm_inference_tutorial,
  title        = {LLM Inference Tutorial: Build a Modified NGC PyTorch Container},
  author       = {{Swiss National Supercomputing Centre (CSCS)}},
  year         = {2025},
  howpublished = {\url{https://docs.cscs.ch/tutorials/ml/llm-inference/#build-a-modified-ngc-pytorch-container}},
  note         = {Accessed: 2025-08-27}
}

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors