1. State of the Art

Current layout of components generated by FC2K is based on Kepler's module layout. The reason for such layout is based on historical choices - related to Kepler being a main workflow engine. Python based actors (being an extension to already existing solution) exploit the same layout. It means, parts of the Python code depend strongly on components created by FC2K.

1.1. Generated code

FC2K generates:

  • Kepler actor
  • Python actor
  • Resources used by both: Python and Kepler actors 
    • libActor.so
      • embeds user code
      • provides Fortran/C wrapper that reads/write IDSes
      • stored in $KEPLER/imas/lib64
    • Actor.exe
      • executable that is used to run actor in standalone mode
        • debugging
        • any kind of batch jobs (e.g. MPI)
    • Other: actor XML parameters, etc. (in case of Python actors these files must be located close to Python module)

1.2. Why Python actor uses $KEPLER ?

  • Python script loads libActor.so  to run user's (Fortran or CPP) code
  • libActor.so is stored in $KEPLER/imas/lib64 directory
    • lib64  is Kepler's default place to look for libraries to be loaded
    • adding library path to LD_LIBRARY_PATH  could be an alternative BUT in case of complex workflows (100+ actors) it can be extremely long  - potential risk of exceeding system limits
  • Python module loads code parameters based on $KEPLER variable

2. Separation of actors 

2.1. Raw idea

Separation of actors, at least in theory, should be easy. It would be enough:

  • To move actor resources out of the $KEPLER
  • To make resources "visible" by both : Kepler and Python actors

but we have also consider many other aspects...

2.2. Open points

Problems to be solved:

  • How to assure compatibility with current version of IMAS?
    • Binaries are build for particular IMAS version 
    • If user switch IMAS version, version of resources should be also changed
  • How to assure compatibility of Kepler or Python workflow with given set of actors
    • Actor A could have different ports (API) in workflow version X and Y
      • This is currently achieved by actor release procedure and maintenance of ETS workflow (release of ETS workflow)
      • ETS workflow + visualisation scripts + Kepler release + actor release + autoGUI release make a package that works.
  • How to make libraries "discoverable" by both actors?
    • LD_LIBRARY_PATH ? 
    • absolute path to libActor.so  ?
    • Any other mechanism?
  • How to design layout of directories to address before-mentioned points?  

3. Proposed solutions

3.1. Independent Python/Kepler actor installations

In this scenario I assume complete separation of Kepler and Python. FC2K should allow choosing whether we want to generate actor for given workflow platform: Python, Kepler, something else in the future. Once platform is selected, FC2K generates source code and "Weighted Companion Cube" - one that contains all the native codes, libraries, etc. - for given platform.

3.1.1. Kepler

Due to the fact that Kepler will be used for ETS only (probably) it seems like there is no much sense in redesigning the whole idea of actor generation.

3.1.2. Python

I suggest moving towards Python packages that are self contained packages consisting of:

  • python wrapper
  • shared libraries
  • standalone binary (MPI based execution)
  • default parameters

Each package should be either installed inside user's virtual environment or should be available via wrapper module - this way, it will be possible to provide users with any version of actor and Python workflow will be composed of loosely coupled actors.

Sample structure of the actor can be found here: /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple 

Inside, we have a sample structure of Python module with native code: actor_demo  and other_actor  - both inside src directory

.
|-- src                         - source code of the actor (native code)
|-- testing                     - scripts used for testing the solution
|-- venv-27                     - virtual environment
`-- workspace                   - workspace (as described by Bartek)
    |-- 3.23.2                  - version of the IMAS (one that was used while building an actor)
    |   |-- actor_demo          - actor directory (this is not Python package)
    |   |   |-- 1.0             - version of the actor
    |   |   |   `-- actor_demo  - this is Python package (that contains module)
    |   |   |-- 1.1
    |   |   |   `-- actor_demo
    |   |   `-- 1.2
    |   |       `-- actor_demo
    |   `-- other_actor         - this is another actor
    |       |-- 1.0
    |       |   `-- other_actor
    |       |-- 1.1
    |       |   `-- other_actor
    |       `-- 1.2
    |           `-- other_actor
    `-- 3.24.0
        |-- actor_demo
        |   |-- 1.0
        |   |   `-- actor_demo
        |   |-- 1.1
        |   |   `-- actor_demo
        |   `-- 1.2
        |       `-- actor_demo
        `-- other_actor
            |-- 1.0
            |   `-- other_actor
            |-- 1.1
            |   `-- other_actor
            `-- 1.2
                `-- other_actor

In this scenario, actor is treated as package. It means, we can either install it using pip command or we can access it via wrapper module by setting some artificial access point for all the actors - PYTHON_WORKSPACE. This approach provides huge flexibility when it comes to distributing actors. It's very easy to create virtual environment and install actors inside it.

3.1.3. Source code - actor

Source code directory contains two sample actors. This is a prototype of what FC2K will generate while building an actor.

src/
|-- compile.sh             - helper script for building actors (explained later)
|-- actor_demo             - actor: actor_demo
|   |-- demo.c.template    - this is the native code - one that actor provides
|   |-- MANIFEST.in        - this is the list of files that are part of the Python package (used by pip)
|   |-- setup.py.template  - this is the description of the package (used by pip)
|   `-- wrapper.py         - this is the wrapper - it exports Python like interface to native code
`-- other_actor            - actor: other actor
    |-- demo.c.template
    |-- MANIFEST.in
    |-- setup.py.template
    `-- wrapper.py

3.1.4. Building the actors

With the structure above, we can easily create actors for different IMAS releases. It will automatically create all the components of the Python packages

# This is the place that will be used as a workspace
mkdir -p workspace

# In case we have something inside, we can remove it
rm -rf workspace/*

# We have to make sure that PYTHON_WORKSPACE points to existing location
setenv PYTHON_WORKSPACE `pwd`/workspace

# Let's start with default settings
module purge
module load cineca
module load imasenv

# Now, we can simulate the process of releasing different actors
# with different version numbers
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2

./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd

# Now, let's switch the environment to different version
# of IMAS
module purge
module load cineca
module load imasenv/3.23.2/ual/4.1.5/1.2

# And let's rebuild the actors
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2

./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd

once we have completed the build process, the workspace will look like this:

workspace/
|-- 3.23.2
|   |-- actor_demo
|   |   |-- 1.0
|   |   |   |-- actor_demo
|   |   |   |   |-- __init__.py
|   |   |   |   |-- libcore.so
|   |   |   |   `-- wrapper.py
|   |   |   |-- MANIFEST.in
|   |   |   `-- setup.py
|   |   |-- 1.1
|   |   |   |-- actor_demo
|   |   |   |   |-- __init__.py
|   |   |   |   |-- libcore.so
|   |   |   |   `-- wrapper.py
|   |   |   |-- MANIFEST.in
|   |   |   `-- setup.py
|   |   `-- 1.2
|   |       |-- actor_demo
|   |       |   |-- __init__.py
|   |       |   |-- libcore.so
|   |       |   `-- wrapper.py
|   |       |-- MANIFEST.in
|   |       `-- setup.py
|   `-- other_actor
|       |-- 1.0
|       |   |-- MANIFEST.in
|       |   |-- other_actor
|       |   |   |-- __init__.py
|       |   |   |-- libcore.so
|       |   |   `-- wrapper.py
|       |   `-- setup.py
|       |-- 1.1
|       |   |-- MANIFEST.in
|       |   |-- other_actor
|       |   |   |-- __init__.py
|       |   |   |-- libcore.so
|       |   |   `-- wrapper.py
|       |   `-- setup.py
|       `-- 1.2
|           |-- MANIFEST.in
|           |-- other_actor
|           |   |-- __init__.py
|           |   |-- libcore.so
|           |   `-- wrapper.py
|           `-- setup.py
`-- 3.24.0
    |-- actor_demo
    |   |-- 1.0
    |   |   |-- actor_demo
    |   |   |   |-- __init__.py
    |   |   |   |-- libcore.so
    |   |   |   `-- wrapper.py
    |   |   |-- MANIFEST.in
    |   |   `-- setup.py
    |   |-- 1.1
    |   |   |-- actor_demo
    |   |   |   |-- __init__.py
    |   |   |   |-- libcore.so
    |   |   |   `-- wrapper.py
    |   |   |-- MANIFEST.in
    |   |   `-- setup.py
    |   `-- 1.2
    |       |-- actor_demo
    |       |   |-- __init__.py
    |       |   |-- libcore.so
    |       |   `-- wrapper.py
    |       |-- MANIFEST.in
    |       `-- setup.py
    `-- other_actor
        |-- 1.0
        |   |-- MANIFEST.in
        |   |-- other_actor
        |   |   |-- __init__.py
        |   |   |-- libcore.so
        |   |   `-- wrapper.py
        |   `-- setup.py
        |-- 1.1
        |   |-- MANIFEST.in
        |   |-- other_actor
        |   |   |-- __init__.py
        |   |   |-- libcore.so
        |   |   `-- wrapper.py
        |   `-- setup.py
        `-- 1.2
            |-- MANIFEST.in
            |-- other_actor
            |   |-- __init__.py
            |   |-- libcore.so
            |   `-- wrapper.py
            `-- setup.py

As you can see, each unique directory pointed by IMAS_VERSION/ACTOR_NAME/ACTOR_VERSION contains self sufficient Python package. This package can be either accessed using helper scripts (Python loaded for the actor) or can be installed using pip.

3.1.5. Virtual environments

Virtual environment provides user based separation of Python environment from the system one. There are multiple choices when it comes to "virtualization" of Python environment:

In this sample, I will use one that is available as Python package.

3.1.5.1. Installation of actor

In this section, I will install actor (version 1.0) execute it and then, I will upgrade actor to version (1.1) and execute it once again.

During the test, I will use very simple test code

'''
We are importing both actors inside env.
'''

from actor_demo.wrapper import actor_demo
from other_actor.wrapper import other_actor

''' Once code is loaded, we can access native functions via Python based wrapper code '''
actor_demo()
other_actor()

wrapper is the name of the module inside both actors. Each actor exports different Python function: actor_demo in case of actor_demo actor, and other_actor in case of other_actor.

# Initialise virtual environment 
> virtualenv --no-site-packages -p /gw/switm/python/2.7/bin/python2.7 venv-27
> source venv-27/bin/activate.csh

# Install actors with version 1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.0

# Now, let's use the actors in the workflow
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
    Hello from the native function - 'fun': 3.24.0 - 1.0
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
    Hello from the native function - 'thecode': 3.24.0 - 1.0
Hello World!


# If we decide to use different version of the code, it's still possible
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.1
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.1
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
    Hello from the native function - 'fun': 3.24.0 - 1.1
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
    Hello from the native function - 'thecode': 3.24.0 - 1.1
Hello World!

This way, it's possible to create custom work environments where various actors can be installed, reinstalled, where we can mix different versions of actors. However, due to pip related limitations we can have only one actor at the time inside virtual environment.

3.1.6. Accessing actors via wrappers

In this scenario, we provide place with all the actors  - WORKSPACE - and via the wrappers we allow user to use different versions of actor. The structure of workspace directory remains the same. We have to make sure we point to this place using environment variable

# Make sure that Python based wrapper will be able to load actor's code
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace

for each actor we need a wrapper

'''
actor_demo_loader.py
'''

import os
import sys
import platform
import imp

def import_actor(actor_name='actor_demo', version=None):

  workspace = os.getenv('PYTHON_WORKSPACE')

  if workspace is None:
    raise Exception(
      'It looks like PYTHON_WORKSPACE is not set')

  python_ver = platform.python_version_tuple()
  sys_arch = platform.machine()

  actor_path = os.path.join(workspace, os.getenv('IMAS_VERSION'), actor_name, version, actor_name)
  print('Importing actor: \n ' + actor_path)

  if not os.path.isdir(actor_path):
    raise Exception(
      'Directory {} does not exists, you have probably not generated any actor'.format(
        actor_path))

  sys.path.append(actor_path)
  fp, pathname, description = imp.find_module('wrapper', [actor_path])
  _imas_module = imp.load_module('wrapper', fp, pathname, description)

  globals().update(_imas_module.__dict__)
  del _imas_module

With this, we can use actors inside regular Python environment (actors are never installed inside site-packages).

> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace

# I am running workflow with version 1.0 of both actors
#
> python testing/test-wrapper-1.0.py
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/actor_demo/1.0/actor_demo
- I am a native code of the actor. It's name is 'actor_demo'.
    Hello from the native function - 'fun': 3.23.2 - 1.0
Hello World!
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/other_actor/1.0/other_actor
- I am a native code of the actor. It's name is 'other_actor'.
    Hello from the native function - 'thecode': 3.23.2 - 1.0
Hello World!

# Now, let's try to run workflow with version 1.1
#
> python testing/test-wrapper-1.1.py
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/actor_demo/1.1/actor_demo
- I am a native code of the actor. It's name is 'actor_demo'.
    Hello from the native function - 'fun': 3.23.2 - 1.1
Hello World!
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/other_actor/1.1/other_actor
- I am a native code of the actor. It's name is 'other_actor'.
    Hello from the native function - 'thecode': 3.23.2 - 1.1
Hello World!

However, this solution has small issue. It requires explicit selection of the module. It means, it's not quite a Python way of loading modules. Note the difference between test-wrapper-1.0.py 

import actor_demo_loader
import other_actor_loader

actor_demo_loader.import_actor(version='1.0')
actor_demo_loader.actor_demo()

other_actor_loader.import_actor(version='1.0')
other_actor_loader.other_actor()

and test-wrapper-1.1.py 

import actor_demo_loader
import other_actor_loader

actor_demo_loader.import_actor(version='1.1')
actor_demo_loader.actor_demo()

other_actor_loader.import_actor(version='1.1')
other_actor_loader.other_actor()

as you can see, we have to explicitly select version of the actor. An alternative approach would be importing actors by embedding version as a part of package (sort of: from actor_demo.1_0.actor_demo import wrapper). Anyway, there is no simple way of making the very same workflow to be compatible with different versions of actor without altering workflow's code, environment variables, whatever the way of selecting the version we choose. 

3.2. Final directory structure

Below you can find final suggestion for the Python based layout

3.25.0                                                                       - IMAS version
`-- coreprofiles2distsources                                                 - name of the actor (package for PIP)
    `-- 1.0.0                                                                - version of the actor (as specified in FC2K's project)
        |-- coreprofiles2distsources                                         - name of the actor (as visible inside Python)
        |   |-- __init__.py                                                  - makes an actor a module
        |   |-- code_parameters
        |   |   |-- cp_default.xml
        |   |   |-- cp_user.xml
        |   |   `-- cp_schema.xsd
        |   |-- native_wrapper                                               - all the native elements required by actor
        |   |   |-- bin
        |   |   |   |-- coreprofiles2distsourcesF.exe                        - standalone execution
        |   |   |   `-- coreprofiles2distsourcesF_opt.exe
        |   |   |-- build
        |   |   |-- lib
        |   |   |   |-- def
        |   |   |   |   `-- libcore_profiles_2_distribution_sources.a        - (do we need this one?)
        |   |   |   |-- libcoreprofiles2distsources_def.so                   - shared libraries
        |   |   |   |-- libcoreprofiles2distsources_opt.so
        |   |   |   |-- libcoreprofiles2distsources.so
        |   |   |   `-- opt
        |   |   |       `-- libcore_profiles_2_distribution_sources.a        - (do we need this one?)
        |   |   |-- lib_ext
        |   |   |   |-- def
        |   |   |   `-- opt
        |   |   |-- Makefile                                                 - Makefile for building all the native based part
        |   |   `-- src
        |   |       |-- fortrantools.f90                                     - Fortran based wrappers
        |   |       |-- FortranWrap.f90
        |   |       |-- RWTools.f90
        |   |       `-- standalone.f90
        |   `-- wrapper.py                                                   - this Python code contains Python based actor
        |-- MANIFEST.in                                                      - required by PIP
        |-- README.md                                                        - required by PIP
        |-- setup.cfg                                                        - required by PIP
        `-- setup.py                                                         - required by PIP

3.3. PyAL modifications

PyAL modifications

Alternative approach: One common installation of actor resources






  • No labels

6 Comments

  1. Unknown User (olivier.hoenen@ipp.mpg.de)

    On the second solution (3.2) I believe that DD version is not the only piece of info that matters to separate versions of libs, AL should be considered too (as additional level, or combined somehow with DD).

    1. Bartosz Palak

      $ACTIVE_WORKSPACE/imas/$IMAS_VERSION/AL/$AL_VERSION/$COMPILER_VENDOR/$COMPILER_VERSION 

      hmm... it begins to be very complex... 

      1. Unknown User (olivier.hoenen@ipp.mpg.de)

        Why would compiler info be needed? 

        1. Bartosz Palak

          OK, if currently Kepler actors dont require such information it means it is not necessary 

  2. Unknown User (olivier.hoenen@ipp.mpg.de)

    3.2 is lacking information on name+signature of the function in the library, so it should either store this information somehow together with lib (complexify the follow-up stages of generating the Kepler/Python codes), or Kepler and Python codes should be stored together with lib (benefit from 3.2 compared to 3.1 is then less obvious to me). 

    1. Bartosz Palak

      Solution 3.1 (yet) not describes how to ensure IMAS ↔ set of actors compatibility