• Created by Unknown User (tzok), last modified on 08 Nov 2018

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

1.1. Introduction

Containerization is a method to isolate computing environment (operating system, data, software, configurations, etc.) into reusable packages called images. These are binary files which can easily be transferred between different machines. Images are instantiated into running Containers by a Container Engine. From user perspective, working with the Container is similar to working with another operating system on the shared hardware resources. This resembles what Virtual Machines provide. However, there are important technical differences between Container and Virtual Machine technologies. Mainly, any number of Containers can be executed by a single Container Engine on a single operating system. In case of Virtual Machines, each runs its own operating system with its own layer of hardware abstraction. In consequence, Containers are in general quicker to start and easier to operate.

1.2. Docker

The main Containerization technology used now is Docker. It consists of image file format description, the Container Engine and other components. It is available for Linux, macOS and Windows.

For more information, check Docker overview

1.3. uDocker

uDocker is "a basic user tool to execute simple docker containers in user space without requiring root privileges".

For more information, check indigo-dc/udocker

1.4. Docker vs uDocker

Important: While uDocker has mostly identical commands and parameters as Docker, there are some minor differences:

  • Docker: 
    • docker create: prepare a container from an image
    • docker start: start a prepared container
    • docker run: prepare a new container and start it in a single step
    • docker exec: execute commands on a started container
  • uDocker: 
    • udocker create: prepare a container from an image
    • udocker run: execute commands on an existing container or prepare a new container and execute command on it
    • There is no udocker start and udocker exec!

In consequence, the typical usage for Docker is combination of docker run and optionally docker exec. For uDocker, the typical usage is a combination of udocker create and udocker run.

Please be aware not to mix these usage scenarios.

uDocker is slower to prepare a container, so incorrect usage of udocker run may end up in starting multiple containers and each operation taking a long time.

1.5. Goal

The goal of IMAS Docker creation is to prepare a Docker image containing full IMAS environment with additional tools such as Kepler and FC2K.

1.6. Latest IMAS Docker image

  • IMAS version: 3.20.0
  • UAL version: 3.8.3
  • Kepler present
  • FC2K not present yet
  • Available on the Gateway: ~g2tomz/public/imas-installer-20180921112143.tar.xz

1.7. Important

  1. Below you will find instructions with example commands to run.
  2. Commands to be run on your local machine will be marked with blue color.
  3. Commands to be run on the Gateway will be marked with green color.
  4. Commands to be run in the running container will be marked with red color.
  5. The documentation covers usage of both Docker (running on your own computer) and uDocker (running on the Gateway). You do not have to follow both scenarios.

1.8. Installing container engine

  • Docker on your own computer:

  • uDocker on Gateway:

    mkdir -p ~/.local/bin $ITMWORK/dot-udocker
ln -s $ITMWORK/dot-udocker ~/.udocker
curl https://raw.githubusercontent.com/indigo-dc/udocker/master/udocker.py > ~/.local/bin/udocker
chmod u+rx ~/.local/bin/udocker
echo "tmpdir='$ITMSCRATCH'" > ~/.udocker/udocker.conf
~/.local/bin/udocker install

1.9. Loading Docker image

  • Docker on your own compujter:

    scp login.eufus.eu:~g2tomz/public/imas-installer-20180921112143.tar.xz ./
xzcat imas-installer-20180921112143.tar.xz | docker load

  • uDocker on Gateway:

    xzcat ~g2tomz/public/imas-installer-20180921112143.tar.xz | ~/.local/bin/udocker load

1.10. Demonstration 1: Python script

  • Create a container and start a shell in it:

    • Docker on your own computer:

      docker run --interactive --tty --name imas imas-installer:20180921112143

    • uDocker on Gateway:

      ~/.local/bin/udocker create --name=imas imas-installer:20180921112143
~/.local/bin/udocker run imas

  • In the container shell, execute:

    module load imas
imasdb test
python /home/imas/imas-installer/src/$IMAS_VERSION/ual/$UAL_VERSION/examples/dd-v3/python/put_pf.py

  • Transfer the results to the Gateway:

    • Docker on your own computer:

      docker cp imas:/home/imas/public/imasdb/test/3/0/ids_120001.characteristics /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_120001.datafile /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_120001.tree /tmp/
scp /tmp/ids_120001.* login.eufus.eu:public/imasdb/test/3/0/

    • uDocker on Gateway:

      cp ~/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_120001.* ~/public/imasdb/test/3/0/

  • Check correctness of the results, on Gateway:

    idsdump 12 1 pf_active

1.11. Demonstration 2: Kepler workflow

 

  • Create a container and start a shell in it:

    • Docker on your own computer:

      docker run --interactive --tty --name imas imas-installer:20180921112143

    • uDocker on Gateway:

      ~/.local/bin/udocker create --name=imas imas-installer:20180921112143
~/.local/bin/udocker run imas

  • In the container shell, execute:

    module load imas kepler
module load keplerdir
imasdb test
export USER=imas
kepler -runwf -nogui -user imas /home/imas/simple-workflow.xml

  • Transfer the results to the Gateway:

    • Docker on your own computer:

      docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.characteristics /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.datafile /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.tree /tmp/
scp /tmp/ids_10001.* login.eufus.eu:public/imasdb/test/3/0/

    • uDocker on Gateway:

      cp ~/.udocker/containers/imas/ROOT/home/imas/public/imasdb/test/3/0/ids_10001.* ~/public/imasdb/test/3/0/

  • On the host machine, execute:

    docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.characteristics /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.datafile /tmp/
docker cp imas:/home/imas/public/imasdb/test/3/0/ids_10001.tree /tmp/

  • Check correctness of the results, on Gateway:

    idsdump 1 1 pf_active
  • No labels