Introduction

Containerization is a method to isolate computing environment (operating system, data, software, configurations, etc.) into reusable packages called images. These are binary files which can easily be transferred between different machines. Images are instantiated into running Containers by a Container Engine. From user perspective, working with the Container is similar to working with another operating system on the shared hardware resources. This resembles what Virtual Machines provide. However, there are important technical differences between Container and Virtual Machine technologies. Mainly, any number of Containers can be executed by a single Container Engine on a single operating system. In case of Virtual Machines, each runs its own operating system with its own layer of hardware abstraction. In consequence, Containers are in general quicker to start and easier to operate.

Docker

The main Containerization technology used now is Docker. It consists of image file format description, the Container Engine and other components. It is available for Linux, macOS and Windows.

For more information, check Docker overview

uDocker

uDocker is:

a basic user tool to execute simple docker containers in user space without requiring root privileges

For more information, check indigo-dc/udocker

Docker vs uDocker

Important: While uDocker has mostly identical commands and parameters as Docker, there are some minor differences:

Docker:
- docker create: prepare a container from an image
- docker start: start a prepared container
- docker run: prepare a new container and start it in a single step
- docker exec: execute commands on a started container
uDocker:
- udocker create: prepare a container from an image
- udocker run: execute commands on an existing container or prepare a new container and execute command on it
- There is no udocker start and udocker exec!

In consequence, the typical usage for Docker is combination of docker run and optionally docker exec. For uDocker, the typical usage is a combination of udocker create and udocker run.

Please be aware not to mix these usage scenarios.

uDocker is slower to prepare a container, so incorrect usage of udocker run may end up in starting multiple containers and each operation taking a long time.

Goal

The goal of IMAS Docker creation is to prepare a Docker image containing full IMAS environment with additional tools such as Kepler and FC2K.

Latest IMAS Docker image

IMAS version: 3.28.1
UAL version: 4.8.1
Kepler version: 2.5p5-3.2.0
FC2K version: 4.11.0
Available on the EF Gateway: ~g2tomz/public/imas-fc2k-latest.tar.xz
Available on the ITER node hpc-login02.iter.org: ~zokt/public/imas-fc2k-latest.tar.xz

Important

Below you will find instructions with example commands to run.
Commands to be run on your local machine will be marked with red color.
Commands to be run on the Gateway will be marked with green color.
Commands to be run in the running container will be marked with blue color.
The documentation covers usage of both Docker (running on your own computer) and uDocker (running on the Gateway). You do not have to follow both scenarios.

Installing container engine

Docker on your own computer:
- Follow instructions at Docker for Windows, or
- Follow instructions at Docker for Mac, or
- Follow instructions at Docker for CentOS, Docker for Debian, Docker for Fedora or Docker for Ubuntu

uDocker:

Code Block

language	bash

if grep MARCONI /etc/motd &>/dev/null; then
	WORK=${WORK}
	SCRATCH=${CINECA_SCRATCH}
	stat ${WORK} &>/dev/null || WORK=${HOME}/work
elif grep 'Nord III' /etc/motd &>/dev/null; then
	WORK=/gpfs/projects/$(id -gn)/$(id -un)
	SCRATCH=/gpfs/scratch/$(id -gn)/$(id -un)
else
	WORK=${HOME}/work
	SCRATCH=${HOME}/scratch
fi
 
mkdir -p ~/.local/bin ${SCRATCH}
ln -s ${WORK}/udocker ~/.udocker

wget https://github.com/indigo-dc/udocker/releases/download/1.3.4/udocker-1.3.4.tar.gz
tar zxvf udocker-1.3.4.tar.gz -C ${WORK}/
chmod u+rx ${WORK}/udocker
echo "tmpdir='${SCRATCH}'" > ~/.udocker/udocker.conf

#test udocker exec
${WORK}/udocker/udocker --help

#install
${WORK}/udocker/udocker install

Info

title	uDocker startup errors

If ${WORK}/udocker/udocker command throws errors try to add python3 command before path.

Code Block
module load itm-python/3.10 python3 ${WORK}/udocker/udocker --help python3 ${WORK}/udocker/udocker install

Loading Docker image

Docker on your own computer

Loading Docker image

To load pre-built Docker image:

Code Block

language	bash

xzcat # Linux

scp login.eufus.eu:~g2tomz/public/imas-installer-20180921112143-fc2k-latest.tar.xz ./
xzcat imas-fc2k-latest.tar.xz | docker load

# macOS

scp login.eufus.eu:~g2tomz/public/imas-fc2k-latest.tar.xz ./
docker load < imas-fc2k-latest.tar.xz

uDocker on Gateway:

Code Block

language	bash

${WORK}/udocker/udocker load -i ~g2tomz/public/imas-fc2k-latest.tar.xz

Demonstration 1: Python script

Create a container and start a shell in it:
- Docker on your own computer:
Start a new Docker container
- Code Block
  language bash
  docker run --interactive --tty --name imas imas
-installer
- /fc2k
- uDocker on Gateway:
  Code Block
  language bash
  export PROOT_NO_SECCOMP=1 ${WORK}/udocker/udocker create --name=imas imas/fc2k ${WORK}/udocker/udocker run imas

In the container shell, execute:

Code Block

language	bash

module load imas
imasdb test
python /home/imas/imas-installer/src/$IMAS_VERSION/ual/$UAL_VERSION/examples/dd-v3/python/put_pf.py

On the host machine, execute

cat << EOF > put_pf.py
 
import imas
if __name__ == '__main__':
    ids = imas.ids(1, 1, 1, 1)
    ids.create_env('imas', 'test', '3')
    ids.pf_active.ids_properties.comment = 'Test data'
    ids.pf_active.ids_properties.homogeneous_time = 0
    ids.pf_active.coil.resize(2)
    ids.pf_active.coil[0].name = 'COIL 1A'
    ids.pf_active.coil[1].name = 'COIL 2B'
    number = 10
    ids.pf_active.coil[0].current.data.resize(number)
    ids.pf_active.coil[0].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[0].current.data[i] = 2 * i
        ids.pf_active.coil[0].current.time[i] = i
    number = number + 2
    ids.pf_active.coil[1].current.data.resize(number)
    ids.pf_active.coil[1].current.time.resize(number)
    for i in range(number):
        ids.pf_active.coil[1].current.data[i] = 2 * i + 10
        ids.pf_active.coil[1].current.time[i] = i + number
    ids.pf_active.put()
EOF
 
python put_pf.py

Transfer the results to the Gateway:

Docker on your own computer:

Code Block

language	bash

docker cp imas:/home/imas/public/imasdb/test/3/0/ids_

120001

10001.characteristics /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_

120001

10001.datafile /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_

120001

10001.tree /tmp/ scp /tmp/

Transfer the generated IDSes to the

ids_10001.* login.eufus.eu:public/imasdb/test/3/0/

uDocker on Gateway:
Code Block
language bash

scp /tmp

cp ${WORK}/udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_

120001

10001.*

login.eufus.eu:

~/public/imasdb/test/3/0/

Check correctness of the results, on Gateway:
Code Block
language bash
idsdump $USER test 3 1 1 pf_active

Demonstration 2: Kepler workflow

Create a container and start a shell in it:
- Docker on your own computer:
Start a new Docker container
- Code Block
  language bash
  docker run --interactive --tty --name imas imas/fc2k
- uDocker on Gateway:
  Code Block
  language bash
  ${WORK}/udocker/udocker create --name=imas imas/fc2k ${WORK}/udocker/udocker run imas

Transfer the Kepler workflow into the container

Docker on your own computer:

Code Block
scp login.eufus.eu:~g2tomz/public/simple-workflow.xml ./ docker cp simple-workflow.xml imas

-installer

:simple-workflow.xml

uDocker on Gateway:

Code Block
cp ~g2tomz/public/simple-workflow.xml ~/.udocker/containers/imas/ROOT/home/imas/

In the container shell, execute:

Code Block

language	bash

module load imas kepler
module load keplerdir
imasdb test
export USER=imascp $IMAS_PREFIX/models/mdsplus/ids_model.characteristics public/imasdb/test/3/0/ids_19998.characteristics
cp $IMAS_PREFIX/models/mdsplus/ids_model.datafile public/imasdb/test/3/0/ids_19998.datafile
cp $IMAS_PREFIX/models/mdsplus/ids_model.tree public/imasdb/test/3/0/ids_19998.tree
kepler -runwf -nogui -user imas /home/imas/simple-workflow.xml

On the host machine, execute

Transfer the results to the Gateway:

Docker on your own computer:

Code Block

language	bash

docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.characteristics /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.datafile /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.tree /tmp/
scp /tmp/ids_10001.* login.eufus.eu:public/imasdb/test/

Transfer the generated IDSes to the

3/0/
uDocker on Gateway:
Code Block
language bash

scp

cp

/tmp

${WORK}/udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.*

login.eufus.eu:

~/public/imasdb/test/3/0/

Check correctness of the results, on Gateway:
Code Block
language bash
idsdump $USER test 3 1 1 pf_active

Page tree

Versions Compared

Old Version 2

New Version Current

Key

Introduction

Docker

uDocker

Docker vs uDocker

Goal

Latest IMAS Docker image

Important

Installing container engine

Loading Docker image

Loading Docker image

Demonstration 1: Python script

Demonstration 2: Kepler workflow

Page tree

Page History

Versions Compared

Old Version 2

New Version Current

Key

Introduction

Docker

uDocker

Docker vs uDocker

Goal

Latest IMAS Docker image

Important

Installing container engine

Loading Docker image

Loading Docker image

Demonstration 1: Python script

Demonstration 2: Kepler workflow