...
In this tutorial the local computer used will be Marconi Gateway. Given that SUMI uses 2.7 python version, first we will load the correspondent module
Code Block | ||
---|---|---|
| ||
module load python/2.7.12 |
SUMI depends on two powerfull libraries to perform its tasks. These are Paramiko (data transfer) and SAGA (remote job submission). To install the dependencies we need to download python libraries, but given that we do not have root permissions we will make usage of a virtual environment which will allow to install these libraries locally. For this purpose we will use "virtualenv". Virtualenv It creates a python environment in local folders and allows to create a virtual python environment where we can install libraries locally. To set it up we run the following commands:
Code Block |
---|
mkdir sumi-virtualenv virtualenv sumi-virtualenv |
...
Code Block |
---|
git clone https://albertbsc@bitbucket.org/albertbsc/sumi.git |
...
This will create a local folder named "sumi". To include it in the $PATH environment variable we need to run the software.
...
SUMI requires two configuration job files which contain the information of the jobs to be sumitted and the HPC cluster where we are going to submit. For this we need to create the configuration folder and copy the configuration files jobs.conf and servers.conf from sumi/conf/ directory.
Code Block | ||
---|---|---|
| ||
mkdir $HOME/.sumi
cp sumi/conf/*.conf $HOME/.sumi |
...
Now, we are ready to run SUMI. Execute the option "-h" to see all the
Code Block |
---|
$ sumi -h usage: sumi.py [-h] [-r] [-u UP] [-d DOWN] [-m MACHINE [MACHINE ...]] [-j JOBS [JOBS ...]] Submission Manager for IMAS optional arguments: -h, --help show this help message and exit -r, --run Run the configured job on a specific cluster -u UP, --upload UP Upload local file -d DOWN, --download DOWN Download remote file -m MACHINE [MACHINE ...], --machine MACHINE [MACHINE ...] List of machines were to submit -j JOBS [JOBS ...], --job JOBS [JOBS ...] List of jobs to be submitted |
...
Then, SUMI has been installed successfully.
Passwordless configuration with Marconi
SAGA and Paramiko require a passwordless connection to work.
Copy your public key to the remote host
Code Block | ||
---|---|---|
| ||
ssh-copy-id username@login.marconi.cineca.it |
Configure sample job and big Marconi connection
...
Download the source code from BitBucket
https://bitbucket.org/albertbsc/sumi/src/master/
Export the variable $SUMI_DIR with the path to SUMI and include the bin directory inside the PATH variable
#!bash
export SUMI_DIR=/path/to/sumi
export PATH=$PATH:$SUMI_DIR/bin
chmod +x $SUMI_DIR/bin/sumi
If you want these variables to persist includes the previous lines in your .bashrc file.
Passwordless configuration
SAGA and Paramiko require a passwordless connection to work.
Copy your public key to the remote host
ssh-copy-id -i ~/.ssh/id_rsa.pub username@example.com
Remote execution
There can be different machines and different jobs specified. Every time that we run SUMI we have to specify to which supercomputer are we submitting to and which of the configured jobs are we going to submit. Otherwise SUMI does not run anything and is conservative. To specify the job and machine we have to use the parameters -j and -m respectively. For example to run a given job named testjob and a machine testmachine we should execute the following command:
sumi -r -j testjob -m testmachine
Be aware that SUMI is case sensitive when specifying the names.
...