...
Introduction
Remote submission of IMAS HPC workflows may become a problem given the requirement for an IMAS environment installed as well as the wide amount of supercomputers available. To approach these issues, an approach with virtualized environment with IMAS installed and a remote submission system has been designed.
Getting started
This tutorial describes the steps to submit an example uDocker IMAS workflow workflow image to a remote supercomputer. To do so we will make usage of uDocker and SUMI (SUbmission Manager for IMAS).
...
The following tutorial has been tested in with the following machines.
- ITER cluster @ITER
- Marconi @Cineca
- Marconi Gatway @Cineca
- Eagle @PSNC
This tutorial follows the next steps
- Install SUMI on the Gateway.
- Passwordless connection with Marconi.
- Install uDocker on Marconi.
- SCP Docker image from Gateway to Marconi and create container
- Test image on Marconi.
- Configure and submit workflow, then check output on GW.
Install SUMI
...
SUMI is a tool capable of submitting jobs to remote HPC clusters and upload and retrieve data from them.
Install SUMI on Marconi Gateway
This subsection describes how to install SUMI on Marconi Gateway. If you want to install it on ITER cluster, please move to the following subsection.
In this tutorial the local computer used will be Marconi Gateway. Given that SUMI uses 2.7 python version, first we will load the correspondent module
...
SUMI depends on two powerfull libraries to perform its tasks. These are Paramiko (data transfer) and SAGA (remote job submission). To install the dependencies we need to download python libraries, but given that we do not have root permissions we will make usage of a virtual environment which will allow to install these libraries locally. For this purpose we will use "virtualenv". It creates a python environment in local folders and allows to create a virtual python environment where we can install libraries locally. To set it up we run the following commands:
| Code Block |
|---|
mkdir sumi-virtualenv virtualenv sumi-virtualenv |
...
In the case of TCSH shells ( as Gateway)
| Code Block | ||
|---|---|---|
| ||
source sumi-virtualenv/bin/activate.csh |
In the case of Bash shells (most used shells)
| Code Block |
|---|
source sumi-virtualenv/bin/activate |
Our terminal prompt will now show the folder name in front of out username in the following way:
| Code Block |
|---|
[sumi-virtualenv] <g2user@s65 |
To install the dependencies, now we can run the "pip" command which will install the python libraries in our local virtualenv
| Code Block | ||
|---|---|---|
| ||
pip install saga-python==0.50.01 paramiko |
Once the dependencies have been installed, we can download and configure SUMI. To retrieve the code, clone the code from the repository
| Code Block |
|---|
wget git clone -qO- https://albertbsc@bitbucketbitbucket.org/albertbsc/sumi/downloads/sumi.git-0.1.0.tar.gz | tar xvz |
This will create a local folder named "sumi". To include it in the $PATH environment variable we need to run the software.
...
SUMI requires two configuration job files which contain the information of the jobs to be sumitted and the HPC cluster where we are going to submit. For this we need to create the configuration folder and copy the configuration files jobs.conf and servers.conf from sumi/conf/ directory.
| Code Block | ||
|---|---|---|
| ||
mkdir $HOME/.sumi
cp sumi/conf/*.conf $HOME/.sumi |
Now, we are ready to run SUMI. Execute the option "-h" to see all the
| Code Block |
|---|
$ sumi -h
usage: sumi.py [-h] [-r] [-u UP] [-d DOWN] [-m MACHINE [MACHINE ...]]
[-j JOBS [JOBS ...]]
Submission Manager for IMAS
optional arguments:
-h, --help show this help message and exit
-r, --run Run the configured job on a specific cluster
-u UP, --upload UP Upload local file
-d DOWN, --download DOWN
Download remote file
-m MACHINE [MACHINE ...], --machine MACHINE [MACHINE ...]
List of machines were to submit
-j JOBS [JOBS ...], --job JOBS [JOBS ...]
List of jobs to be submitted |
Then, SUMI has been installed successfully.
Install SUMI on ITER cluster
This subsection describes how to install SUMI on ITER cluster. If you want to install it on Marconi Gateway, please move to the previous subsection.
In this tutorial the local computer used will be Marconi Gateway. Given that SUMI uses 2.7 python version, first we will load the correspondent module
| Code Block | ||
|---|---|---|
| ||
module load python/2.7.15 |
SUMI depends on two powerfull libraries to perform its tasks. These are Paramiko (data transfer) and SAGA (remote job submission). To install the dependencies we need to download python libraries, but given that we do not have root permissions we will make usage of a virtual environment which will allow to install these libraries locally. For this purpose we will use "virtualenv". It creates a python environment in local folders and allows to create a virtual python environment where we can install libraries locally. To set it up we run the following commands:
To install the dependencies, now we can run the "pip" command which will install the python libraries in our local virtualenv
| Code Block |
|---|
pip install saga-python==0.50.01 --user pip install paramiko --user |
Once the dependencies have been installed, we can download and configure SUMI. To retrieve the code, clone the code from the repository
| Code Block |
|---|
wget -qO- https://bitbucket.org/albertbsc/sumi/downloads/sumi-0.1.0.tar.gz | tar xvz |
This will create a local folder named "sumi". To include it in the $PATH environment variable we need to run the software.
For Bash shells
| Code Block | ||
|---|---|---|
| ||
export PATH=$PATH:$PWD/sumi/bin/ |
SUMI requires two configuration job files which contain the information of the jobs to be sumitted and the HPC cluster where we are going to submit. For this we need to create the configuration folder and copy the configuration files jobs.conf and servers.conf from sumi/conf/ directory.
| Code Block | ||
|---|---|---|
| ||
mkdir $HOME/.sumi mkdir $HOME/.sumi cp sumi/conf/*.conf $HOME/.sumi |
Now, we are ready to run SUMI. Execute the option "-h" to see all the
| Code Block |
|---|
$ sumi -h
usage: sumi.py [-h] [-r] [-u UP] [-d DOWN] [-m MACHINE [MACHINE ...]]
[-j JOBS [JOBS ...]]
Submission Manager for IMAS
optional arguments:
-h, --help show this help message and exit
-r, --run Run the configured job on a specific cluster
-u UP, --upload UP Upload local file
-d DOWN, --download DOWN
Download remote file
-m MACHINE [MACHINE ...], --machine MACHINE [MACHINE ...]
List of machines were to submit
-j JOBS [JOBS ...], --job JOBS [JOBS ...]
List of jobs to be submitted |
Then, SUMI has been installed successfully on ITER cluster.
Passwordless configuration with Marconi
To use SUMI we need a passwordless connection. This means that Marconi the targeted HPC machine needs to have the public key of our account at Gateway on the list of allowed keys. If we don't have a ~/.ssh/id_rsa.pub file we have to generate our keys
...