...

This tutorial describes the steps to submit an example uDocker IMAS workflow workflow image to a remote supercomputer. To do so we will make usage of uDocker and SUMI (SUbmission Manager for IMAS).

These work on different sides of the system: local machine and remote HPC system:

• Connect from a local computer to a remote cluster to submit my workflow: IMAS
• Bring IMAS environment to supercomputer heterogenous systems: uDocker image

This tutorial assumes that the user has a function machine with a distribution of GNU/Linux installed.

The following tutorial has been tested in the following machines.

• Marconi @Cineca
• Marconi Gatway @Cineca
• Eagle @PSNC

This tutorial follows the next steps

1. Install SUMI on the Gateway

2. Configure sample job and big Marconi connection

3. Test connection with Marconi

4. Install uDocker on Marconi

5. Test image on Marconi

...

2. Submit workflow

3. Retrieve data

4. Visualize output on GW

    

Install SUMI on the Gateway

SUMI is a tool capable of submitting jobs to remote HPC clusters and upload and retrieve data from them.

In this tutorial the local computer used will be Marconi Gateway. Given that SUMI uses 2.7 python version, first we will load the correspondent module

module load python/2.7.12

SUMI depends on two powerfull libraries to perform its tasks. These are Paramiko (data transfer) and SAGA (remote job submission). To install the dependencies we need to download python libraries, but given that we do not have root permissions we will make usage of a virtual environment which will allow to install these libraries locally. For this purpose we will use "virtualenv". Virtualenv creates a python environment in local folders and allows to create a virtual python environment where we can install libraries locally. To set it up we run the following commands:

mkdir sumi-virtualenv

virtualenv sumi-virtualenv

Once the virtualenv folder has been configured, we can load the environment.

In the case of TCSH shells (as Gateway)

source sumi-virtualenv/bin/activate.csh

In the case of Bash shells (most used shells)

source sumi-virtualenv/bin/activate

Our terminal prompt will now show the folder name in front of out username in the following way:

[sumi-virtualenv] <g2user@s65

To install the dependencies, now we can run the "pip" command which will install the python libraries in our local virtualenv

pip install saga-python==0.50.01 paramiko

Once the dependencies have been installed, we can download and configure SUMI. To retrieve the code, clone the code from the repository

git clone https://albertbsc@bitbucket.org/albertbsc/sumi.git

This will create a local folder named "sumi". To include it in the $PATH environment variable we need to run the software.

For TCSH shell systems (as Gateway)

setenv PATH $PATH\:$HOME/sumi/bin

For Bash shells

export PATH=$PATH:$PWD/sumi/bin/

Now, we are ready to run SUMI. Execute the option "-h" to see all the

...

We will make usage of uDocker. uDocker is based on Docker but it does not require root permissions. Therefore, it does not have some features as TCP/IP sniffing or other action that require root privileges. uDocker will allow to simulate an environment with root permissions (UID = 0) and install IMAS inside.

Therefore, the very first step is installing uDocker. uDokcer will be set up once and used many times

...