Hypatia is the University’s new HPC. Some more details can be found here: https://www.st-andrews.ac.uk/high-performance-computing/
The HPC uses slurm for queueing jobs and resource management, and has several different partitions (some info is given upon logging into Hypatia). Use slurm to gain access to the compute nodes. Don’t execute large operations on the login node!
Installing anaconda
>> install-conda
>> conda create -n <ENV_NAME>
Then activate the environment:
>> conda activate <ENV_NAME>
Running JupyterLab
Start by acquiring resources on a compute node and returning an interactive shell. With slurm, you can do this with the ‘srun’ command. In the following example, we acquire 1GB of memory for 1 hour on the small-short partition.
>> srun --pty -t 0-01:00 -p small-short --mem=1GB /bin/bash
Assuming you are working in an anaconda environment (and you have installed jupyter), you need to activate it on the compute node, where <ENV_NAME> is the name of your environment in /software/conda/<USERNAME>/conda/envs. The following can be written into a simple bash script for regular use.
>> conda activate <ENV_NAME>
Find the hostname of the compute node:
>> hostname -i
Now run JupyterLab with no browser:
>> jupyter-lab --no-browser --ip=<hostname -i>
The terminal should return the address of the Jupyter server:
>> Jupyter Server is running at:
>> http:// <hostname -i> :XXXX/lab
where XXXX is often, but not always, 8888 (make sure you check this). In a second terminal window, you need to connect to the Jupyter server:
>> ssh -L 8080:<hostname -i>:XXXX <USERNAME>@hypatia.st-andrews.ac.uk
Open a browser window and navigate to
localhost:8080
You may need to enter a password or a token (the latter can be found in the terminal where you loaded Jupyter).
Example shell scripts for interactive compute node use
Ensure these are executables: chmod +x
#!/bin/bash
# Run this on the login node to acquire compute node resources and return interactive shell
# Usage: ./run_srun.sh <time> <memory> <partition>
# Example (1 hour 24GB on small-short): ./run_srun.sh 0-1:00 24G small-short
TIME=$1
MEMORY=$2
PARTITION=$3
srun --pty -t "$TIME" -p "$PARTITION" --mem="$MEMORY" /bin/bash
#############
#!/bin/bash
# Run this on compute node to launch JupyterLab
source ~/.bashrc
conda activate <ENV_NAME>
hostvar=$(hostname -i)
echo $hostvar
jupyter-lab --no-browser --ip=$hostvar
#############
Storage
Shared scratch storage in /sharedscratch/<username>
I have had success with using Cyberduck for interactive file transfer.
ERA5 data
Read README.txt first!
ERA5 data are mostly on a 1° grid once-daily at 00 UTC (unless otherwise stated) on a series of pressure levels (which should be consistent across variables).
There are also various python files which are used to download/update each dataset – feel free to copy these and use them to acquire any additional ERA5 data yourself.
Most of the data are downloaded from the CDS as GRIB files and converted to netCDF using xarray.Dataset.to_netcdf with cfgrib.