Info

title	Quote of the day

ANAKIN: Is it possible to learn this power?
PALPATINE: Not from a Jedi.

Star Wars: Revenge of the Sith

This tutorial is about working with IMAS in an isolated environment (containers), which you can instantiate anytime, copy and reuse with little effort on different machines. After this tutorial you will gain knowledge of:

...

To download an image from Docker Hub, use: docker pull <image-name>
To list available images, use: docker images
To remove image, use: docker rmi <image-name>

Example 1. Pulling an image

Image Added

Containers

To run a container, use: docker run <image-name> <command>
To list containers, use: docker ps
To copy a file from/into a running container, use: docker cp <file-1> <file-2>
To execute a command in a running container, use: docker exec <container-id> <command>
To remove container, use: docker rm <container-id>

Info

title	Flags

Each command supports additional flags passed along the main arguments. Please make sure to check docker help <command> for more information.

The most often used flags are:

--name for docker run to specify friendly name for the container
-i, --interactive and -t, --tty for docker run and docker exec when you want to work in an interactive shell inside of the container

Example 2. Starting a container

Image Added

Exercises

Checking /etc/os-release of different containers

...

Search for Ubuntu image: singularity search ubuntu
Download latest image: singularity pull library://library/default/ubuntu
Notice that the image is in fact a file downloaded to your local directory: ls -l ubuntu_latest.sif
Run the image: ./ubuntu_latest.sif
Notice that you are mapped to the same user as in the host machine, run whoami and id
Notice that you are seeing your own $HOME directory from the container: ls -l
Check what system are you currently running: cat /etc/os-release

Info

Note that while you are inside container (ubuntu_latest.sif) executed by singularity your system will report as

Code Block
> cat /etc/os-release NAME="Ubuntu" VERSION="18.10 (Cosmic Cuttlefish)" ID=ubuntu ID_LIKE=debian ... ...

while Marconi node will present itself as

Code Block
> cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ... ...

uDocker

uDocker was created in H2020 project INDIGO-DataCloud. It uses Docker images natively, but provides a userspace tool to run and manage containers. Because it does not require superuser privileges, it is also a good fit as a container tool to be used in HPCs. uDocker also supports MPI calculations and provides ways to access host's GPU.

...

uDocker is contained in a single Python file making it easy to deploy in distributed systems. To install it on Gateway execute:

Code Block

language	bash

setenvexport WORK =/pfs/work/$USER
setenv SCRATCH /pfs/scratch/$USER

mkdir -p ~/.local/bin ${WORK}/dot-udocker ${SCRATCH}
ln -s ${WORK}/dot-udocker ~/.udocker
 
curlwget https://raw.githubusercontentgithub.com/indigo-dc/udocker/master/releases/download/1.3.4/udocker.py > ~/.local/bin/udocker-1.3.4.tar.gz
tar zxvf udocker-1.3.4.tar.gz -C ${WORK}/
chmod u+rx ~/.local/bin${WORK}/udocker
 
echo "tmpdir='#test udocker exec
${SCRATCH}'" > ~/.WORK}/udocker/udocker.conf
~/.local/bin --help
 
#install
${WORK}/udocker/udocker install

Info

title	uDocker on other machines

The same steps can be repeated on other machines, including HPC or your local computer. Just change WORK and SCRATCH variables in the first two lines. Note, that SCRATCH will be used during import of new images and WORK will be used to store images and files on all running containers. Choose them wisely to accommodate possibly big amount of data.

Info

title	uDocker startup errors

If ${WORK}/udocker/udocker command throws errors try to add python3 command before path.

Code Block
module load itm-python/3.10 python3 ${WORK}/udocker/udocker --help python3 ${WORK}/udocker/udocker install

Exercises

Start uDocker container

Download latest Ubuntu image: ~/.local/bin/${WORK}/udocker/udocker pull ubuntu
Verify available images: ~/.local/bin${WORK}/udocker/udocker images images
Create a container: ~/.local/bin${WORK}/udocker/udocker create create --name=udocker-ubuntu ubuntu
List running containers: ~/.local/bin/udocker ps ${WORK}/udocker/udocker ps
Run an interactive shell: ~/.local/bin${WORK}/udocker/udocker run run udocker-ubuntu /bin/bash
Check user: whoami
Check operating system: cat /etc/os-release
Exit from the container: exit
Check operating system at Gateway: cat /etc/os-release
Delete the container: ~/.local/bin/udocker rm udocker-ubuntu ${WORK}/udocker/udocker rm udocker-ubuntu

Info

As in case of Singularity, here, you can spot the difference as well. As long as you are inside uDocker based container you will see

Code Block
> cat /etc/os-release NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian ... ...

while Gateway node reports itself as

Code Block
> cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ... ...

IMAS image

IMAS environment is available as a Docker image. There are two flavors of the image: batch and GUI. Each image has a predefined environment, so unlike on Gateway, you do not need to load any modules or call imasdb to get started. Both images them are exported as archives and made available on the Gateway.

...

To load the images, please run:

Code Block
~/.local/bin${WORK}/udocker/udocker load -i ~g2tomz/public/imas-fc2k-latest.tar.xz ~/.local/bin${WORK}/udocker/udocker load -i ~g2tomz/public/imas-gui-latest.tar.xz

...

Make sure that the images are loaded: ~/.local/bin${WORK}/udocker/udocker images images
Create container: ~/.local/bin${WORK}/udocker/udocker create create --name=imas imas/fc2k:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run an interactive shell: ~/.local/bin${WORK}/udocker/udocker run run imas /bin/bash

Prepare and run a Python script which will create IDS pf_active in shot 1 and run 1:

Code Block

cat << EOF > put_pf.py
  
import imas
if __name__ == '__main__':
    ids = imas.ids(1, 1, 1, 1)
    ids.create_env('imas', 'test', '3')
    ids.pf_active.ids_properties.comment = 'Test data'
    ids.pf_active.ids_properties.homogeneous_time = 0
    ids.pf_active.coil.resize(2)
    ids.pf_active.coil[0].name = 'COIL 1A'
    ids.pf_active.coil[1].name = 'COIL 2B'
    number = 10
    ids.pf_active.coil[0].current.data.resize(number)
    ids.pf_active.coil[0].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[0].current.data[i] = 2 * i
        ids.pf_active.coil[0].current.time[i] = i
    number = number + 2
    ids.pf_active.coil[1].current.data.resize(number)
    ids.pf_active.coil[1].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[1].current.data[i] = 2 * i + 10
        ids.pf_active.coil[1].current.time[i] = i + number
    ids.pf_active.put()
EOF
  
python put_pf.py

You can now exit container shell. Note, that the container is not executing anything, but it persists and all files are kept in ~/.udocker. Copy the generated pulsefile directly into your Gateway's collection of pulsefiles:
cp ~${WORK}/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ~/public/imasdb/test/3/0/
Verify that the IDS pf_active has been created: idsdump $USER test 3 1 1 pf_active

Running Kepler workflow in a container

Create container: ~/.local/bin${WORK}/udocker/udocker create --name=imas-gui imas/gui:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run the default application (VNC server) with port mapping: ~/.local/bin${WORK}/udocker/udocker run run --publish 15901:5901 imas-gui
In another terminal, open VNC viewer, connect to localhost:15901 and and use imas as as the password: vncviewer localhost:15901
You will see XFCE4 Openbox desktop environment. Open terminal and run: kepler, with Kepler loading automatically (please wait until it is ready).
Design an example workflow like the one below (it reads , which will read pf_active IDS from shot 1 and run 1 as created by previous step):
Image Removed:
Image Added
Run the workflow and notice that it fails due to lack of pulsefile with required content. This is because in the previous exercise you were running in the imas container, isolated from the imas-gui one.
You can now do one of the following:
1. Create the pulsefile again
  1. Run in another terminal in the Gateway: ${WORK}/udocker/udocker run imas-gui /bin/bash
  2. Repeat step 4 from previous exercise
2. Copy pulsefile between containers
  1. Run cp ${WORK}/udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ${WORK}/udocker/containers/imas-gui/ROOT/home/imas/public/imasdb/test/3/0/
Start the workflow again. The expected result:

Page tree

Versions Compared

Old Version 9

New Version Current

Key

Example 1. Pulling an image

Containers

Example 2. Starting a container

Exercises

Checking /etc/os-release of different containers

uDocker

Exercises

Start uDocker container

IMAS image

Running Kepler workflow in a container

Page tree

Page History

Versions Compared

Old Version 9

New Version Current

Key

Example 1. Pulling an image

Containers

Example 2. Starting a container

Exercises

Checking /etc/os-release of different containers

uDocker

Exercises

Start uDocker container

IMAS image

Running Kepler workflow in a container