Info

title	Quote of the day

ANAKIN: Is it possible to learn this power?
PALPATINE: Not from a Jedi.

Star Wars: Revenge of the Sith

This tutorial is about working with IMAS in an isolated environment (containers), which you can instantiate anytime, copy and reuse with little effort on different machines. After this tutorial you will gain knowledge of:

...

To download an image from Docker Hub, use: docker pull <image-name>
To list available images, use: docker images
To remove image, use: docker rmi <image-name>

Example 1. Pulling an image

Image Added

Containers

To run a container, use: docker run <image-name> <command>
To list containers, use: docker ps
To copy a file from/into a running container, use: docker cp <file-1> <file-2>
To execute a command in a running container, use: docker exec <container-id> <command>
To remove container, use: docker rm <container-id>

Info

title	Flags

Each command supports additional flags passed along the main arguments. Please make sure to check docker help <command> for more information.

The most often used flags are:

--name for docker run to specify friendly name for the container
-i, --interactive and -t, --tty for docker run and docker exec when you want to work in an interactive shell inside of the container

Example 2. Starting a container

Image Added

Exercises

Checking /etc/os-release of different containers

...

uDocker is contained in a single Python file making it easy to deploy in distributed systems. To install it on Gateway execute:

Code Block

language	bash

setenvexport WORK =/pfs/work/$USER
setenv SCRATCH /pfs/scratch/$USER

mkdir -p ~/.local/bin ${WORK}/dot-udocker ${SCRATCH}
ln -s ${WORK}/dot-udocker ~/.udocker
curl 
wget https://rawgithub.githubusercontent.com/indigo-dc/udocker/releases/masterdownload/udocker.py > ~/.local/bin/udocker
chmod u+rx ~/.local/bin/udocker
echo "tmpdir='${SCRATCH}'" > ~/.udocker/udocker.conf
~/.local/bin1.3.4/udocker-1.3.4.tar.gz
tar zxvf udocker-1.3.4.tar.gz -C ${WORK}/
chmod u+rx ${WORK}/udocker
 
#test udocker exec
${WORK}/udocker/udocker --help
 
#install
${WORK}/udocker/udocker install

Info

title	uDocker on other machines

The same steps can be repeated on other machines, including HPC or your local computer. Just change WORK and SCRATCH variables in the first two lines. Note, that SCRATCH will be used during import of new images and WORK will be used to store images and files on all running containers. Choose them wisely to accommodate possibly big amount of data.

...

Info

title

...

uDocker

...

startup errors

If ${WORK}/udocker/udocker command throws errors try to add python3 command before path.

Code Block
module load itm-python/3.10 python3 ${WORK}/udocker/udocker --help python3 ${WORK}/udocker/udocker install

Exercises

Start uDocker container

Download latest Ubuntu image: ${WORK}/udocker/udocker pull ubuntu
Verify available images: ${WORK}/udocker/udocker images
Create a container: ${WORK}/udocker/udocker create --name=udocker-ubuntu ubuntu
List running containers: ${WORK}/udocker/udocker ps
Run an interactive shell: ${WORK}/udocker/udocker run
Download latest Ubuntu image: ~/.local/bin/udocker pull ubuntu
Verify available images: ~/.local/bin/udocker images
Create a container: ~/.local/bin/udocker create --name=udocker-ubuntu ubuntu
List running containers: ~/.local/bin/udocker ps
Run an interactive shell: ~/.local/bin/udocker run udocker-ubuntu /bin/bash
Check user: whoami
Check operating system: cat /etc/os-release
Exit from the container: exit
Check operating system at Gateway: cat /etc/os-release
Delete the container: ~/.local/bin${WORK}/udocker/udocker rm rm udocker-ubuntu

IMAS image

IMAS environment is available as a Docker image. There are two flavors of the image: batch and GUI. Each image has a predefined environment, so unlike on Gateway, you do not need to load any modules or call imasdb to get started. Both images them are exported as archives and made available on the Gateway.

Loading images to uDocker

To load the images, please run:

Code Block
~/.local/bin/udocker load -i ~g2tomz/public/imas-fc2k-latest.tar.xz ~/.local/bin/udocker load -i ~g2tomz/public/imas-gui-latest.tar.xz

Info
Each step takes a bit longer than 1 minute without any progress indication, so please wait until it is finished.

Exercises

Running Python script to create IDSes

Info

As in case of Singularity, here, you can spot the difference as well. As long as you are inside uDocker based container you will see

Code Block
> cat /etc/os-release NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian ... ...

while Gateway node reports itself as

Code Block
> cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ... ...

IMAS image

IMAS environment is available as a Docker image. There are two flavors of the image: batch and GUI. Each image has a predefined environment, so unlike on Gateway, you do not need to load any modules or call imasdb to get started. Both images them are exported as archives and made available on the Gateway.

Loading images to uDocker

To load the images, please run:

Code Block
${WORK}/udocker/udocker load -i ~g2tomz/public/imas-fc2k-latest.tar.xz ${WORK}/udocker/udocker load -i ~g2tomz/public/imas-gui-latest.tar.xz

Info
Each step takes a bit longer than 1 minute without any progress indication, so please wait until it is finished.

Exercises

Running Python script to create IDSes

Make sure that the images are loaded: ${WORK}/udocker/udocker images
Create container:${WORK}/udocker/udocker create --name=imas imas/fc2k:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run an interactive shell: ${WORK}/udocker/udocker run imas /bin/bash

Prepare and run a Python script which will create IDS pf_active in shot 1 and run 1:

Code Block
cat << EOF > put_pf.py import imas if __name__ == '__main__': ids = imas.ids(1, 1, 1, 1) ids.create_env('imas', 'test', '3'

Make sure that the images are loaded: ~/.local/bin/udocker images
Create container: ~/.local/bin/udocker create --name=imas imas/fc2k:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run an interactive shell: ~/.local/bin/udocker run imas /bin/bash

Prepare and run a Python script which will create IDS pf_active in shot 1 and run 1:

Code Block

cat << EOF > put_pf.py
  
import imas
if __name__ == '__main__':
    ids = imas.ids(1, 1, 1, 1)
    ids.create_env('imas', 'test', '3')
    ids.pf_active.ids_properties.comment = 'Test data'
    ids.pf_active.ids_properties.homogeneous_time = 0
    ids.pf_active.coil.resize(2)
    ids.pf_active.coil[0].name = 'COIL 1A'
    ids.pf_active.coil[1].name = 'COIL 2B'
    number = 10
    ids.pf_active.coil[0].current.data.resize(number)
    ids.pf_active.coil[0].current.time.resize(number)
    for i in range(number):ids_properties.comment = 'Test data'
    ids.pf_active.ids_properties.homogeneous_time = 0
    ids.pf_active.coil.resize(2)
    ids.pf_active.coil[0].current.data[i]name = 2 * i
    'COIL 1A'
    ids.pf_active.coil[01].current.time[i] = iname = 'COIL 2B'
    number = number + 210
    ids.pf_active.coil[10].current.data.resize(number)
    ids.pf_active.coil[10].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[10].current.data[i] = 2 * i + 10
        ids.pf_active.coil[10].current.time[i] = i
    number = number + number2
    ids.pf_active.put()
EOF
  
python put_pf.py

You can now exit container shell. Note, that the container is not executing anything, but it persists and all files are kept in ~/.udocker. Copy the generated pulsefile directly into your Gateway's collection of pulsefiles:
cp ~/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ~/public/imasdb/test/3/0/
Verify that the IDS pf_active has been created: idsdump $USER test 3 1 1 pf_active

Running Kepler workflow in a container

.coil[1].current.data.resize(number)
    ids.pf_active.coil[1].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[1].current.data[i] = 2 * i + 10
        ids.pf_active.coil[1].current.time[i] = i + number
    ids.pf_active.put()
EOF
  
python put_pf.py

You can now exit container shell. Note, that the container is not executing anything, but it persists and all files are kept in ~/.udocker. Copy the generated pulsefile directly into your Gateway's collection of pulsefiles:
cp ${WORK}/udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ~/public/imasdb/test/3/0/
Verify that the IDS pf_active has been created: idsdump $USER test 3 1 1 pf_active

Running Kepler workflow in a container

Create container: ${WORK}/udocker/udocker create --name=imas-gui imas/gui:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run the default application (VNC server) with port mapping: ${WORK}/udocker/udocker run --publish 15901:5901 imas-gui
In another terminal, open VNC viewer, connect to localhost:15901 and use imas as the password: vncviewer localhost:15901
You will see Openbox desktop environment, with Kepler loading automatically (please wait until it is ready).
Design an example workflow like the one below, which will read pf_active IDS from shot 1 and run 1:
Image Added
Run the workflow and notice that it fails due to lack of pulsefile with required content. This is because in the previous exercise you were running in the imas container, isolated from the imas-gui one.
You can now do one of the following:
1. Create the pulsefile again
  1. Run in another terminal in the Gateway: ${WORK}/udocker/udocker run imas-gui /bin/bash
  2. Repeat step 4 from previous exercise
2. Copy pulsefile between containers
  1. Run cp ${WORK}/udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ${WORK}/udocker/containers/imas-gui/ROOT/home/imas/public/imasdb/test/3/0/
Start the workflow again.
Create container: ~/.local/bin/udocker create --name=imas-gui imas/gui:3.24.0-4.2.0-2.5p4-3.0.5-4.6.5
Run the default application (VNC server) with port mapping: ~/.local/bin/udocker run --publish 15901:5901 imas-gui
In another terminal, open VNC viewer, connect to localhost:15901 and use imas as the password: vncviewer localhost:15901
You will see XFCE4 desktop environment. Open terminal and run: kepler
Design an example workflow like the one below (it reads pf_active from shot 1 and run 1 as created by previous step):
Image Removed
The expected result:

Page tree

Versions Compared

Old Version 11

New Version Current

Key

Example 1. Pulling an image

Containers

Example 2. Starting a container

Exercises

Checking /etc/os-release of different containers

Exercises

Start uDocker container

IMAS image

Loading images to uDocker

Exercises

Running Python script to create IDSes

IMAS image

Loading images to uDocker

Exercises

Running Python script to create IDSes

Running Kepler workflow in a container

Running Kepler workflow in a container

Page tree

Page History

Versions Compared

Old Version 11

New Version Current

Key

Example 1. Pulling an image

Containers

Example 2. Starting a container

Exercises

Checking /etc/os-release of different containers

Exercises

Start uDocker container

IMAS image

Loading images to uDocker

Exercises

Running Python script to create IDSes

IMAS image

Loading images to uDocker

Exercises

Running Python script to create IDSes

Running Kepler workflow in a container

Running Kepler workflow in a container