Using Andromeda

Generating an ssh key

On Windows you can install the “OpenSSH Client” under “Optional Features” in Windows Settings. You can then use a PowerShell window to run the ssh commands. From a Mac, you can use ssh from the Terminal application (found under /Applications/Utilities). Linux users can run ssh from any terminal or console.

If you don’t already have a key, you must generate an ssh key and send the public half (never send the private key) to hpc@etal.uri.edu when you first sign up. Students should CC their Advisor on this email. If you are not the PI on the project, please CC them in the request. On the command line (PowerShell, Terminal, etc), you should have a command called ssh-keygen to create a key if you do not already have a key (when prompted, press Enter to use the default file name). Set a passphrase on this key. You will be prompted for this passphrase when you connect. This will generate two files: the private key ~/.ssh/id_rsa and the public key ~/.ssh/id_rsa.pub. Only send the id_rsa.pub file in the email.

NOTE: On macOS you can either attach the file, or use the command cat ~/.ssh/id_rsa.pub | pbcopy to copy the contents to the clipboard. You may need to use ⌘-Shift-Period to toggle being able to see files/directories starting with a “.”.

Transferring Data

Small files

Use scp/sftp to transfer small amounts of data to/from the cluster. The recommended client for this is Cyberduck. If you are using a graphical interface for transferring data, you might want to bookmark the paths you use most frequently. See this ▶️video for setting up Cyberduck.

Remote transfers

To transfer data from remote sites, you will need to pull the data from the remote site, as our system is not available remotely. So, to transfer to /data/somegroup/, choose one of (in order of preference):

rsync -av remotesystem:/remote/path/ ./localpath/
scp -r remotesystem:/remote/path ./localpath
wget -O ./localpath https://remotesystem/url
curl -O https://remotesystem/url

Or, to copy data off, one of (in order of preference):

rsync -av ./localpath/ remotesystem:/remote/path/
scp -r ./localpath remotesystem:/remote/path

See the man pages for how to use each of rsync/scp/wget/curl.

Managing Data

If your program needs to create temporary files, they should do this in /tmp. For some programs the way to do this is to set the environment variable via export TMPDIR="/tmp/". Note that most compute nodes have ~1TB available there, and it is not shared across nodes. Also make sure your script cleans up anything it puts there.

Important items

MPI

SLURM has some built-in support for working with OpenMPI. Information on working with all MPI implementations can be found at SLURM’s MPI page.

Interactive mode

See below for details of using interactive/debugging sessions.

Environment variables

SLURM by default will copy your existing environment from the login shell. While this may be helpful for testing, for reproducible results, it is usually better if your script sets all the variables and loads any modules directly. To get this behavior, use sbatch --export=NONE.

sbatch Parameters

NOTE: You can always check the manual on the system for more options using the command man sbatch.

There is tab-completion support for all of the new commands, so you can type sbatch --<tab><tab> to see all of the options. You can also review the documentation below the online SLURM man pages). SLURM’s documentation includes a guide to converting from other schedulers here (PDF) if your software has examples of using it in other HPC environments.

Useful parameters are:

-t HH:MM:SS

This specifies the maximum wall-clock time the process will be given before it is terminated.

--nodes 2 --ntasks-per-node=36

This specifies to allocate 2 nodes and 36 processors (cores) per node. For most simple programs, this should be --nodes=1. (See below for how to change this effectively.)

--cpus-per-task=4

This specifies to allocate 4 processors (cores) of a node. For most simple programs, this should be used with a --nodes=1 parameter.

--mem=20G

If you need more than 6GB of RAM per task, you can request it with something like--mem=250G or --mem=510G.

--mem-per-cpu=6G

Per-cpu version of above.

--mail-type=BEGIN,END,FAIL,TIME_LIMIT_{50,80,90}

Email you when the job starts, completes or fails. TIME_LIMIT_80 alerts when 80% of the wall time is reached. 50 and 90% are also available.

--mail-user=username@example.edu

Specify the email address to send notifications to. You could also put this in ~/.forward

-o slurm-%j.out

Specify where the output from the program should go (stdout only if -e is also specified). Use %j for job id or %A_%a for array jobs.

-e slurm-%j.out

Specify where the error output from the program should go (stderr only).

--array=1-100%10

Create an array job. In this case, the array indices would be 1-100, with a limit of 10 jobs running concurrently.

--export=VAR=val

Add an environment variables to the job.

--export=NONE

Start the job with a “clean” environment, not with the variables in the current shell.

--exclusive

Node exclusive (See below for when to use this.)

--exclusive=user

When using less than one node (for example -c 1), use --exclusive=user to keep the jobs on as few nodes as possible.

--chdir=/start/path

Specify a path to start working in. Normally jobs are started in $HOME. This sets $SLURM_SUBMIT_DIR. Many scripts start with cd $SLURM_SUBMIT_DIR to make the directory sbatch was run from be the directory for the rest of the script.

--wrap="command"

Specify a single command to run as a batch job. This removes the need to have a script, if you specify all of the sbatch parameters on the command line.

--qos="normal"

Specify a Quality of Service to use (for groups with high priority access, see below).

Choosing a job size

1. Does the program you are using support MPI (the documentation will say so)? If yes, then you can use multiple nodes, and want to use them in exclusive mode. You will also likely use ‘srun’ or ‘mpirun’ to run the program, depending on how the program was compiled. This may take a bit of experimentation. You need to also look further to see if it supports threads. If so (sometimes called “hybrid” mode), then you want to use $SLURM_CPUS_ON_NODE as the number of threads you specify to the program (unless the program auto-detects this, or has another recommendation). If it does not support threads, then use --ntasks-per-core=1.
2. If your program does not use MPI (the typical case), then determine if it supports threads. If it does, then you should again use exclusive mode with --nodes=1 and $SLURM_CPUS_ON_NODE to maximize use of the node.
3. If the program does not mention using MPI or threads, it may mention using OpenMP (Note this is different than OpenMPI). OpenMP can also use threads (--exclusive --nodes=1), but in this case you want to export OMP_NUM_THREADS=$SLURM_CPUS_ON_NODE. Depending on the application, you may wish to balance --ntasks-per-node and --cpus-per-task to achieve the best performance.
4. If none of the above apply, then your program likely only runs on one core, and you don’t need to specify node, cpu or task option (the default is one core). In this case, if you are running the same code against independent data, you may want to consider array jobs.

Using multiple nodes

You must check if your program is capable of using multiple cores and/or multiple nodes to do computation before requesting resources with the --nodes option. If your program uses the MPI libraries, than you should be able to adjust these parameters within the constraints of that program. For example, some programs require that the number of concurrent processes is even or a multiple of some other number. If your program allows you to specify a number of threads, then you may want to set -c 36 (the number of cores in most of our nodes, although some have 24) and use $SLURM_NTASKS_PER_NODE to pass that number to your program.

Exclusive node use

• Your job is going to use all of the cores in a node. In this case, you can pass $SLURM_CPUS_ON_NODE to your program for the number of threads
• If your job is going to use less than the number of cores on a node, but is going to use almost almost all of the RAM on that node.
• You want to use less than the number of cores on a node, but are measuring the performance of the job and want to ensure consistent results.

Large memory jobs

Environment

While jobs are running on the computation nodes, there are environment variables available. The most important of these are below:

SLURM_JOB_ID

The ID of the job. You can use squeue to see the nodes allocated to your job.

SLURM_SUBMIT_DIR

Directory from which sbatch was called.

SLURM_ARRAY_TASK_ID

Unique ID assigned to this instance of the job. (Array jobs only.)

Job Status

squeue

Gives a list of jobs in the queue. Some options are:

--long or -l

Long format (more details).

--states=RUNNING or -t RUNNING

List only running jobs

--states=PENDING or -t PENDING

List only pending jobs

-r

Expand array jobs (only for pending jobs).

-u $USER

Show only your jobs.

scontrol show job

Shows the details of a currently running job.

sacct

Show job history for your user. Use -l for all details.

Canceling a job

X11 forwarding

Using modules

avail

Shows a list of software that is available to use. Some software is in groups, so you can do module list bio/ to see the list of biology related software. Use module list without any modifiers to see the complete list, although there will be some duplication.

load

Used to setup the environment for a particular software. E.g., module load BamTools will make the bamtools command available. Note that some modules may not be loaded side-by-side, such as those compiled with different dependencies. Sometimes multiple versions of a package will be available. You can specify an exact version, such as module load BamTools/2.4.0-foss-2016b

unload

Unload a particular module.

list

Shows the list of loaded modules.

show

Shows how a module will change the environment.

purge

Reset the environment.

module load xxx/yyy-foss-2018b
./prog1 # depends only on xxx
module purge
module load www/zzz-foss-2016b
./prog2 # depends only on www

Scheduling Policy