...
Independent Python/Kepler actor installations
TBD (MICHAL)
One common installation of actor resources
Idea
- To keep all (common) actor resources for both: Python and Kepler actors in one place (directory)
- To allow switching between "workspaces" (user defined sets of actors)
- To load actor libraries only for current version of IMAS (if actor is IMAS dependent)
Layout of actor directory
TBD (the same as previously?)
IMAS Workspace
In this scenario I assume complete separation of Kepler and Python. FC2K should allow choosing whether we want to generate actor for given workflow platform: Python, Kepler, something else in the future. Once platform is selected, FC2K generates source code and "Weighted Companion Cube" - one that contains all the native codes, libraries, etc. - for given platform.
Kepler
Due to the fact that Kepler will be used for ETS only (probably) it seems like there is no much sense in redesigning the whole idea of actor generation.
Python
I suggest moving towards Python packages that are self contained packages consisting of:
- python wrapper
- shared libraries
- standalone binary (MPI based execution)
- default parameters
Each package should be either installed inside user's virtual environment or should be available via wrapper module - this way, it will be possible to provide users with any version of actor and Python workflow will be composed of loosely coupled actors.
Sample structure of the actor can be found here: /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple
Inside, we have a sample structure of Python module with native code: actor_demo and other_actor - both inside src directory
| Code Block |
|---|
.
|-- src |
| Code Block |
workspaces/ |-- workspaceX |-- workspaceY `-- workspaceZ |-- common | |-- actor1 | | `-- lib | | `-- libActor1.so | |-- actor2 | | `-- lib | | `-- libActor2.so source code of the |actor `(native code) |-- testing lib | |-- libActor1.so -> ../actor1/lib/libActor1.so | - scripts used for testing the `-- libActor2.so -> ../actor2/lib/libActor2.so solution |-- venv-27 `-- imas |-- 3.23.2 - virtual environment `-- workspace | |-- actorA | | `-- lib workspace (as described by Bartek) |-- 3.23.2 | `-- libActorA.so - |version of the |-- actorB |IMAS (one that was used while building an actor) | `|-- lib actor_demo | - |actor directory (this is not `-- libActorB.soPython package) | | |-- actorC 1.0 | | `-- lib version of the actor | | | `-- libActorC.so actor_demo - this is Python package (that |contains module) `-- lib | | ||-- 1.1 | |-- libActorA.so -> ../actorA/lib/libActorA.so | `-- actor_demo | | |`-- libActorB.so -> ../actorB/lib/libActorB.so1.2 | | `-- libActorC.so -> ../actorC/lib/libActorC.so `-- 3.24.0 actor_demo | `-- other_actor - this is |--another actorAactor | | `-- lib1.0 | | `-- libActorA.soother_actor | |-- actorB1.1 | | `-- libother_actor | `-- 1.2 | `-- libActorB.so other_actor `-- 3.24.0 |-- actorDactor_demo | | `-- lib1.0 | | `-- libActorD.soactor_demo | `|-- lib1.1 | | |`-- libActorA.so -> ../actorA/lib/libActorA.soactor_demo | `-- 1.2 | |`-- libActorB.so -> ../actorB/lib/libActorB.so actor_demo `-- other_actor `|-- libActorD1.so0 -> ../actorD/lib/libActorD.so |
Workspace layout:
- directory
common- keeps all actors with no IMAS dependencies - directory
imas/$IMAS_VERSION- keeps all actors build for given version of IMAS - directories
common/libandimas/$IMAS_VERSION/lib- keeps links to libraries (to simplifyLD_LIBRARY_PATH)
Actor generation:
- Generated code (wrapper) will be saved under
$ACTIVE_WORKSPACE/imas/$IMAS_VERSION/<actor_name>
Switching workspace
LD_LIBRARY_PATH = $ACTIVE_WORKSPACE/common/lib + $ACTIVE_WORKSPACE/imas/$IMAS_VERSION/lib- java.lib.path = $LD_LIBRARY_PATH
- Kepler: rm target/* ? ant compile?
Scripts:
- list workspaces
- switch workspace
- remove workspace
- create workspace
Open points
...
| `-- other_actor
|-- 1.1
| `-- other_actor
`-- 1.2
`-- other_actor
|
In this scenario, actor is treated as package. It means, we can either install it using pip command or we can access it via wrapper module by setting some artificial access point for all the actors - PYTHON_WORKSPACE. This approach provides huge flexibility when it comes to distributing actors. It's very easy to create virtual environment and install actors inside it.
Source code - actor
Source code directory contains two sample actors. This is a prototype of what FC2K will generate while building an actor.
| Code Block |
|---|
src/
|-- compile.sh - helper script for building actors (explained later)
|-- actor_demo - actor: actor_demo
| |-- demo.c.template - this is the native code - one that actor provides
| |-- MANIFEST.in - this is the list of files that are part of the Python package (used by pip)
| |-- setup.py.template - this is the description of the package (used by pip)
| `-- wrapper.py - this is the wrapper - it exports Python like interface to native code
`-- other_actor - actor: other actor
|-- demo.c.template
|-- MANIFEST.in
|-- setup.py.template
`-- wrapper.py |
Building the actors
With the structure above, we can easily create actors for different IMAS releases. It will automatically create all the components of the Python packages
| Code Block |
|---|
# This is the place that will be used as a workspace
mkdir -p workspace
# In case we have something inside, we can remove it
rm -rf workspace/*
# We have to make sure that PYTHON_WORKSPACE points to existing location
setenv PYTHON_WORKSPACE `pwd`/workspace
# Let's start with default settings
module purge
module load cineca
module load imasenv
# Now, we can simulate the process of releasing different actors
# with different version numbers
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2
./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd
# Now, let's switch the environment to different version
# of IMAS
module purge
module load cineca
module load imasenv/3.23.2/ual/4.1.5/1.2
# And let's rebuild the actors
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2
./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd |
once we have completed the build process, the workspace will look like this:
| Code Block |
|---|
workspace/
|-- 3.23.2
| |-- actor_demo
| | |-- 1.0
| | | |-- actor_demo
| | | | |-- __init__.py
| | | | |-- libcore.so
| | | | `-- wrapper.py
| | | |-- MANIFEST.in
| | | `-- setup.py
| | |-- 1.1
| | | |-- actor_demo
| | | | |-- __init__.py
| | | | |-- libcore.so
| | | | `-- wrapper.py
| | | |-- MANIFEST.in
| | | `-- setup.py
| | `-- 1.2
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| `-- other_actor
| |-- 1.0
| | |-- MANIFEST.in
| | |-- other_actor
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | `-- setup.py
| |-- 1.1
| | |-- MANIFEST.in
| | |-- other_actor
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | `-- setup.py
| `-- 1.2
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
`-- 3.24.0
|-- actor_demo
| |-- 1.0
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| |-- 1.1
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| `-- 1.2
| |-- actor_demo
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| |-- MANIFEST.in
| `-- setup.py
`-- other_actor
|-- 1.0
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
|-- 1.1
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
`-- 1.2
|-- MANIFEST.in
|-- other_actor
| |-- __init__.py
| |-- libcore.so
| `-- wrapper.py
`-- setup.py |
As you can see, each unique directory pointed by IMAS_VERSION/ACTOR_NAME/ACTOR_VERSION contains self sufficient Python package. This package can be either accessed using helper scripts (Python loaded for the actor) or can be installed using pip.
Virtual environments
Virtual environment provides user based separation of Python environment from the system one. There are multiple choices when it comes to "virtualization" of Python environment:
- https://docs.python.org/3/tutorial/venv.html
- https://docs.conda.io/en/latest/
- https://www.anaconda.com
In this sample, I will use one that is available as Python package.
Installation of actor
In this section, I will install actor (version 1.0) execute it and then, I will upgrade actor to version (1.1) and execute it once again.
During the test, I will use very simple test code
| Code Block |
|---|
'''
We are importing both actors inside env.
'''
from actor_demo.wrapper import actor_demo
from other_actor.wrapper import other_actor
''' Once code is loaded, we can access native functions via Python based wrapper code '''
actor_demo()
other_actor() |
wrapper is the name of the module inside both actors. Each actor exports different Python function: actor_demo in case of actor_demo actor, and other_actor in case of other_actor.
| Code Block |
|---|
# Initialise virtual environment
> virtualenv --no-site-packages -p /gw/switm/python/2.7/bin/python2.7 venv-27
> source venv-27/bin/activate.csh
# Install actors with version 1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.0
# Now, let's use the actors in the workflow
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.24.0 - 1.0
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.24.0 - 1.0
Hello World!
# If we decide to use different version of the code, it's still possible
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.1
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.1
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.24.0 - 1.1
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.24.0 - 1.1
Hello World! |
This way, it's possible to create custom work environments where various actors can be installed, reinstalled, where we can mix different versions of actors. However, due to pip related limitations we can have only one actor at the time inside virtual environment.
Accessing actors via wrappers
In this scenario, we provide place with all the actors - WORKSPACE - and via the wrappers we allow user to use different versions of actor. The structure of workspace directory remains the same. We have to make sure we point to this place using environment variable
| Code Block |
|---|
# Make sure that Python based wrapper will be able to load actor's code
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace |
for each actor we need a wrapper
| Code Block |
|---|
'''
actor_demo_loader.py
'''
import os
import sys
import platform
import imp
def import_actor(actor_name='actor_demo', version=None):
workspace = os.getenv('PYTHON_WORKSPACE')
if workspace is None:
raise Exception(
'It looks like PYTHON_WORKSPACE is not set')
python_ver = platform.python_version_tuple()
sys_arch = platform.machine()
actor_path = os.path.join(workspace, os.getenv('IMAS_VERSION'), actor_name, version, actor_name)
print('Importing actor: \n ' + actor_path)
if not os.path.isdir(actor_path):
raise Exception(
'Directory {} does not exists, you have probably not generated any actor'.format(
actor_path))
sys.path.append(actor_path)
fp, pathname, description = imp.find_module('wrapper', [actor_path])
_imas_module = imp.load_module('wrapper', fp, pathname, description)
globals().update(_imas_module.__dict__)
del _imas_module |
With this, we can use actors inside regular Python environment (actors are never installed inside site-packages).
| Code Block |
|---|
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace
# I am running workflow with version 1.0 of both actors
#
> python testing/test-wrapper-1.0.py
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/actor_demo/1.0/actor_demo
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.23.2 - 1.0
Hello World!
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/other_actor/1.0/other_actor
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.23.2 - 1.0
Hello World!
# Now, let's try to run workflow with version 1.1
#
> python testing/test-wrapper-1.1.py
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/actor_demo/1.1/actor_demo
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.23.2 - 1.1
Hello World!
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/other_actor/1.1/other_actor
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.23.2 - 1.1
Hello World! |
However, this solution has small issue. It requires explicit selection of the module. It means, it's not quite a Python way of loading modules. Note the difference between test-wrapper-1.0.py
| Code Block |
|---|
import actor_demo_loader
import other_actor_loader
actor_demo_loader.import_actor(version='1.0')
actor_demo_loader.actor_demo()
other_actor_loader.import_actor(version='1.0')
other_actor_loader.other_actor() |
and test-wrapper-1.1.py
| Code Block |
|---|
import actor_demo_loader
import other_actor_loader
actor_demo_loader.import_actor(version='1.1')
actor_demo_loader.actor_demo()
other_actor_loader.import_actor(version='1.1')
other_actor_loader.other_actor() |
as you can see, we have to explicitly select version of the actor. An alternative approach would be importing actors by embedding version as a part of package (sort of: from actor_demo.1_0.actor_demo import wrapper). Anyway, there is no simple way of making the very same workflow to be compatible with different versions of actor without altering workflow's code, environment variables, whatever the way of selecting the version we choose.
Final directory structure
Below you can find final suggestion for the Python based layout
| Code Block |
|---|
3.25.0 - IMAS version
`-- coreprofiles2distsources - name of the actor (package for PIP)
`-- 1.0.0 - version of the actor (as specified in FC2K's project)
|-- coreprofiles2distsources - name of the actor (as visible inside Python)
| |-- __init__.py - makes an actor a module
| |-- code_parameters
| | |-- cp_default.xml
| | |-- cp_user.xml
| | `-- cp_schema.xsd
| |-- native_wrapper - all the native elements required by actor
| | |-- bin
| | | |-- coreprofiles2distsourcesF.exe - standalone execution
| | | `-- coreprofiles2distsourcesF_opt.exe
| | |-- build
| | |-- lib
| | | |-- def
| | | | `-- libcore_profiles_2_distribution_sources.a - (do we need this one?)
| | | |-- libcoreprofiles2distsources_def.so - shared libraries
| | | |-- libcoreprofiles2distsources_opt.so
| | | |-- libcoreprofiles2distsources.so
| | | `-- opt
| | | `-- libcore_profiles_2_distribution_sources.a - (do we need this one?)
| | |-- lib_ext
| | | |-- def
| | | `-- opt
| | |-- Makefile - Makefile for building all the native based part
| | `-- src
| | |-- fortrantools.f90 - Fortran based wrappers
| | |-- FortranWrap.f90
| | |-- RWTools.f90
| | `-- standalone.f90
| `-- wrapper.py - this Python code contains Python based actor
|-- MANIFEST.in - required by PIP
|-- README.md - required by PIP
|-- setup.cfg - required by PIP
`-- setup.py - required by PIP |
PyAL modifications
...
Alternative approach: One common installation of actor resources
...