...

This tutorial follows the next steps

1. Install SUMI on the Gateway.
2. Test Passwordless connection with Marconi.
3. Install uDocker on Marconi.
4. SCP Docker image from Gateway to Marconi and create container
5. Test image on Marconi.
6. Configure sample job

7. Submit workflow

8. Retrieve data

9. Test and submit workflow, then check output on GW .

Install SUMI on the Gateway

...

Passwordless configuration with Marconi

SAGA and Paramiko require To use SUMI we need a passwordless connection to work.
Copy your public key to the remote host. This means that Marconi needs to have the public key of our account at Gateway on the list of allowed keys. If we don't have a ~/.ssh/id_rsa.pub file we have to generate our keys

ssh-keygen

To copy our key we make usage of the "ssh-copy-id" command to copy from Gateway to Marconi

ssh-copy-id username@login.marconi.cineca.it

Now, if we stablish an SSH connection, the prompt will not ask for any password and will give us a terminal inside Marconi.

 ssh username@login.marconi.cineca.it

Install uDocker on Marconi

For this step follow the instructions the will be found on the uDocker documentation.

Known issues

When setting up a uDocker image locally be sure that there is enough quota for your user. Otherwise it may crash or show untar related problems as the following when running the udocker load command.

    Error: failed to extract container:
    Error: loading failed

SCP Docker image from Gateway to Marconi and create container

The Docker image is avaiable at Gateway and you can use it on any machine, including your own laptop

ls ~g2tomz/public/imas-installer*

To copy the image from Gateway to Marconi we run scp command

scp ~g2tomz/public/imas-installer-20180921112143.tar.xz-id username@login.marconi.cineca.it:

Once it has been copied we log in Marconi and run the following commands to load the image and create the container

$HOME/.local/bin/udocker load -i imas-installer-20180921112143.tar.xz

$HOME/.local/bin/udocker create --name=imas imas-installer:20180921112143

...

Test image on Marconi

Once the image has been loaded we can interact with it and get a bash inside the container. To interact with the image we can get a bash with the following command

$HOME/.local/bin/udocker run imas /bin/bash

The commands that we are going to run for our example case are the following (same as used in uDocker documentation)

module load imas kepler
module load keplerdir
imasdb test
export USER=imas
kepler -runwf -nogui -user imas /home/imas/simple-workflow.xml

Running these commands inside the terminal will make the workflow start running

Configure and submit workflow, then check output on GW

To configure a job we have to edit the files copied on the

mkdir $HOME/.sumi
cp sumi/conf/*.conf $HOME/.sumi

jobs.conf

The configuration file jobs.conf located at local directory $HOME/.sumi/ contains the configuration for the jobs to be run. The sample configuration file located at $SUMI_DIR/conf/jobs.conf has the following content.
   

    [test]

...

    udocker = udocker.py

...

    arguments =
    cpus = 1
    time = 1
    threads_per_process = 1

servers.conf

The configuration file servers.conf located at local directory $HOME/.sumi/ contains the configuration for the servers where SUMI will connect The sample configuration file located at $SUMI_DIR/conf/servers.conf has the following content.

...

SUMI allows to upload and download files automatically. For this we can assume a directory "mywf" in our remote Marconi directory and another one in our local Gateway account, as well as a mywfresults folder on Gateway. This will

[machinemarconi]
server = login.marconi.cineca.it
user = my_username
manager = slurm
protocol = ssh
upload_files = /afs/eufus.eu/g2itmdev/user/my_username/mywf/*
upload_to = /marconi/home/userexternal/agutierr/mywf/
download_files = /marconi/home/userexternal/agutierr/mywf/test*
download_to = /afs/eufus.eu/g2itmdev/user/g2agutie/mywfresults/

Once the job has been configured we can run it using the following command

sumi -r -j test -m marconi

This will copy the files, run the workflow and retrieve the output results. Once we have them we can check whether the

idsdump 1 1 pf_active

And this will show the correct structre of the IDS generated.

Configure sample job and big Marconi connection

Test connection with Marconi

To use SUMI we need a passwordless connection. This means that Marconi needs to have the public key of our account at Gateway on the list of allowed keys. To copy our key we make usage of the "ssh-copy-id" command

...

     $SUMI_UDOCKER_DIR/udocker.py create --name=imas imas-installer:20180529181447

Known issues

 When setting up a uDocker image locally be sure that there is enough quota for your user. Otherwise it may crash or show untar related problems as the following when running the udocker load command.
    Error: failed to extract container:
    Error: loading failed

Setting up SUMI

SUMI (SUbmission Manager for IMAS) is a tool developed for ITER IMAS (Integrated Modelling & Analysis Suite). SUMI aims to launch jobs remotely to HPC machines with a uDocker installation and it is able to manage data transfer among local machine and remote file system. SUMI allows to configure the connection to different supercomputers and configure jobs to be launched to those machines.

Installing SUMI locally is needed to be able to submit


SUMI depends on two free software libraries. Both work with Python 2.x, specifically versions newer than 2.7.x

...