HPC tricks

Introduction

Some “one-liners” that I use pretty often on an HPC. The goal is to add to this list piecemeal and eventually memorize some of ’em.

Slurm-related

See partition usage by all users

squeue -p partition_name

Using a job array

# This is in a job submission script.
# Values provided to --array become an environment variable, $SLURM_ARRAY_TASK_ID
# Automatically spawns n jobs
# Use the variable as an argument for the main script
SBATCH --array=0-3
...
python script.py --argument1 $SLURM_ARRAY_TASK_ID 

Limiting the number of concurrent jobs spawned by an array

# To be a good boi on a server
# The integer after % determines how many concurrent jobs can run
SBATCH --array=1-1000%5

Updating the number of concurrent jobs in an array

# No longer a good boi on a server
scontrol update JobId=<JobID> ArrayTaskThrottle=<count>

Using a job array with multiple arguments

# Map the array variable to line numbers of another file containing more arguments
SBATCH --array=1-3
LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" file.csv)
arg1=$(echo $LINE | awk -F ',' '{print $1}')
arg2=$(echo $LINE | awk -F ',' '{print $2}')
python script.py --argument1 $arg1 --argument2 $arg2

Rsync-related

Delete many files quickly

# Basically target_dir is synced to empty_dir (so everything is deleted)
# Apparently faster than rm -rf?
rsync -aP --delete empty_dir/ target_dir/

Copy pattern-matching files recursively

# ** searches recursively
# --include and --exclude arguments are processed in order
# option -n for dry-run
rsync -avhm --stats --progress  --include="**/*.csv" --include="**/*.json"  --include="*/"  --exclude="*" path/to/source /path/to/destination

\ (•◡•) /