Introduction
Containerization is a method to isolate computing environment (operating system, data, software, configurations, etc.) into reusable packages called images. These are binary files which can easily be transferred between different machines. Images are instantiated into running Containers by a Container Engine. From user perspectiveuser perspective, working with the Container is similar to working with another operating system on the shared hardware resources. This resembles what Virtual Machines provide. However, there are important technical differences between Container and Virtual Machine technologies. Mainly, any number of Containers can be executed by a single Container Engine on a single operating system. In case of Virtual Machines, each runs its own operating system with its own layer of hardware abstraction. In consequence, Containers are in general quicker to start and easier to operate.
Docker
The main Containerization technology used now is Docker. It consists of image file format description, the Container Engine and other components. It is available for Linux, macOS and Windows.
For more information, check check Docker overview
uDocker
uDocker is ":
a basic user tool to execute simple docker containers in user space without requiring root privileges
...
For more information, check check indigo-dc/udocker
Docker vs uDocker
Important: While uDocker has mostly identical commands and parameters as Docker, there are some minor differences:
...
uDocker is slower to prepare a container, so incorrect usage of udocker run
may end up in starting multiple containers and each operation taking a long time.
Goal
The goal of IMAS Docker creation is to prepare a Docker image containing full IMAS environment with additional tools such as Kepler and FC2K.
Latest IMAS Docker image
- IMAS version: 3.2028.01
- UAL version: 34.8.31
- Kepler presentversion: 2.5p5-3.2.0
- FC2K version: 4.11.0FC2K not present yet
- Available on the EF Gateway:
~g2tomz/public/imas-fc2k-latest.tar.xz
- Available on the ITER node hpc-login02.iter.org:
~zokt/public/imas-installerfc2k-20180921112143latest.tar.xz
Important
- Below you will find instructions with example commands to run.
- Commands to be run on your local machine will be shown on green backgroundmarked with red color.
- Commands to be run on the Gateway will be shown on white backgroundmarked with green color.
- Commands to be run in the running container will be marked with red blue color.
- The documentation covers usage of both Docker (running on your own computer) and uDocker (running on the Gateway). You do not have to follow both scenarios.
Installing container engine
- Docker on your own computer:
- Follow instructions at Docker for Windows, or
- Follow instructions at Docker for Mac, or
- Follow instructions at Docker for CentOS, Docker for Debian, Docker for Fedora or Docker for Ubuntu
uDocker on Gateway:
Code Block language bash if grep MARCONI /etc/motd &>/dev/null; then WORK=${WORK} SCRATCH=${CINECA_SCRATCH} stat ${WORK} &>/dev/null || WORK=${HOME}/work elif grep 'Nord III' /etc/motd &>/dev/null; then WORK=/gpfs/projects/$(id -gn)/$(id -un) SCRATCH=/gpfs/scratch/$(id -gn)/$(id -un) else WORK=${HOME}/work SCRATCH=${HOME}/scratch fi mkdir -p ~/.local/bin $ITMWORK/dot-udocker${SCRATCH} ln -s $ITMWORK/dot-${WORK}/udocker ~/.udocker curl wget https://raw.githubusercontentgithub.com/indigo-dc/udocker/master/udocker.py > ~/.local/bin/udocker/releases/download/1.3.4/udocker-1.3.4.tar.gz tar zxvf udocker-1.3.4.tar.gz -C ${WORK}/ chmod u+rx ~/.local/bin/${WORK}/udocker echo "tmpdir='$ITMSCRATCH${SCRATCH}'" > ~/.udocker/udocker.conf ~/.local/bin #test udocker exec ${WORK}/udocker/udocker --help #install ${WORK}/udocker/udocker install
Info title uDocker startup errors If
${WORK}/udocker/udocker
command throws errors try to addpython3
command before path.Code Block module load itm-python/3.10 python3 ${WORK}/udocker/udocker --help python3 ${WORK}/udocker/udocker install
Loading Docker image
Docker on your own compujtercomputer:
themeCode Block DJango language bash # Linux scp login.eufus.eu:~g2tomz/public/imas-installerfc2k-20180921112143latest.tar.xz ./ xzcat imas-installerfc2k-20180921112143latest.tar.xz | docker load # macOS scp login.eufus.eu:~g2tomz/public/imas-fc2k-latest.tar.xz ./ docker load < imas-fc2k-latest.tar.xz
uDocker on Gateway:
Code Block theme Midnightlanguage bash xzcat${WORK}/udocker/udocker load -i ~g2tomz/public/imas-installerfc2k-20180921112143latest.tar.xz | ~/.local/bin/udocker load
Demonstration 1: Python script
Create a container and start a shell in it:
Docker on your own computer:
Code Block theme DJango language bash docker run --interactive --tty --name imas imas-installer:20180921112143/fc2k
uDocker on Gateway:
Code Block language bash ~/.local/binexport PROOT_NO_SECCOMP=1 ${WORK}/udocker/udocker create --name=imas imas-installer:20180921112143 ~/.local/bin//fc2k ${WORK}/udocker/udocker run imas
In the container shell, execute:
Code Block language bash module load imas imasdb test python /home/imas/imas-installer/src/$IMAS_VERSION/ual/$UAL_VERSION/examples/dd-v3/python/cat << EOF > put_pf.py import imas if __name__ == '__main__': ids = imas.ids(1, 1, 1, 1) ids.create_env('imas', 'test', '3') ids.pf_active.ids_properties.comment = 'Test data' ids.pf_active.ids_properties.homogeneous_time = 0 ids.pf_active.coil.resize(2) ids.pf_active.coil[0].name = 'COIL 1A' ids.pf_active.coil[1].name = 'COIL 2B' number = 10 ids.pf_active.coil[0].current.data.resize(number) ids.pf_active.coil[0].current.time.resize(number) for i in range(number): ids.pf_active.coil[0].current.data[i] = 2 * i ids.pf_active.coil[0].current.time[i] = i number = number + 2 ids.pf_active.coil[1].current.data.resize(number) ids.pf_active.coil[1].current.time.resize(number) for i in range(number): ids.pf_active.coil[1].current.data[i] = 2 * i + 10 ids.pf_active.coil[1].current.time[i] = i + number ids.pf_active.put() EOF python put_pf.py
Transfer the results to the Gateway:
Docker on your own computer:
Code Block theme DJangolanguage bash docker cp imas:/home/imas/public/imasdb/test/3/0/ids_12000110001.characteristics /tmp/ docker cp imas:/home/imas/public/imasdb/test/3/0/ids_12000110001.datafile /tmp/ docker cp imas:/home/imas/public/imasdb/test/3/0/ids_12000110001.tree /tmp/ scp /tmp/ids_12000110001.* login.eufus.eu:public/imasdb/test/3/0/
uDocker on Gateway:
Code Block language bash cp ~${WORK}/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_12000110001.* ~/public/imasdb/test/3/0/
Check correctness of the results, on Gateway:
Code Block language bash idsdump 12 $USER test 3 1 1 pf_active
Demonstration 2: Kepler workflow
Create a container and start a shell in it:
Docker on your own computer:
Code Block theme DJangolanguage bash docker run --interactive --tty --name imas imas-installer:20180921112143/fc2k
uDocker on Gateway:
Code Block language bash ~/.local/bin${WORK}/udocker/udocker create --name=imas imas-installer:20180921112143 ~/.local/bin/udocker run imas/fc2k ${WORK}/udocker/udocker run imas
Transfer the Kepler workflow into the container
Docker on your own computer:
Code Block scp login.eufus.eu:~g2tomz/public/simple-workflow.xml ./ docker cp simple-workflow.xml imas:simple-workflow.xml
uDocker on Gateway:
Code Block cp ~g2tomz/public/simple-workflow.xml ~/.udocker/containers/imas/ROOT/home/imas/
In the container shell, execute:
Code Block language bash module load imas kepler module load keplerdir imasdb test export USER=imascp $IMAS_PREFIX/models/mdsplus/ids_model.characteristics public/imasdb/test/3/0/ids_19998.characteristics cp $IMAS_PREFIX/models/mdsplus/ids_model.datafile public/imasdb/test/3/0/ids_19998.datafile cp $IMAS_PREFIX/models/mdsplus/ids_model.tree public/imasdb/test/3/0/ids_19998.tree kepler -runwf -nogui -user imas /home/imas/simple-workflow.xml
Transfer the results to the Gateway:
Docker on your own computer:
Code Block theme DJango language bash docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.characteristics /tmp/ docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.datafile /tmp/ docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.tree /tmp/ scp /tmp/ids_10001.* login.eufus.eu:public/imasdb/test/3/0/
uDocker on Gateway:
Code Block language bash cp ~${WORK}/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ~/public/imasdb/test/3/0/
Check correctness of the results, on Gateway:
Code Block language bash idsdump $USER test 3 1 1 pf_active