...
Inside, we have a sample structure of Python module with native code: demo.c.actor_demo and other_actor - both inside src directory
| Code Block |
|---|
.
|-- src - source code of the actor (native code)
|-- testing - scripts used for testing the solution
|-- venv-27 - virtual environment
`-- workspace - workspace (as described by Bartek)
`|-- 3.24.023.2 - version of the IMAS (one that was used while building an actor)
| `|-- actor_demo - actor directory (this is not Python package)
| | |-- 1.0 - version of the actor
| | | | `-- actor_demo - this is Python package (that contains module)
| | |-- 1.1
| | | `-- actor_demo
| | `-- 1.12
| | `-- actor_demo
| `-- actor_demo |
In this scenario, actor is treated as package. It means, we can either install it using pip command or we can access it via wrapper module by setting some artificial access point for all the actors - PYTHON_WORKSPACE. This approach provides huge flexibility when it comes to distributing actors. It's very easy to create virtual environment and install actor inside it.
Virtual environments
Virtual environment provides user based separation of Python environment from the system one. There are multiple choices when it comes to "virtualization" of Python environment:
- https://docs.python.org/3/tutorial/venv.html
- https://docs.conda.io/en/latest/
- https://www.anaconda.com
In this sample, I will use one that is available as Python package.
Installation of actor
In this section, I will install actor (version 1.0) execute it and then, I will upgrade actor to version (1.1) and execute it once again.
During the test, I will use very simple test code
| Code Block |
|---|
from actor_demo import wrapper
wrapper.actor_demo() |
wrapper is the name of the file inside actor_demo module - it's an arbitrary choice.
| Code Block |
|---|
# Initialise virtual environment
> virtualenv --no-site-packages -p /gw/switm/python/2.7/bin/python2.7 venv-27
> source venv-27/bin/activate.csh
# Install actor with version 1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.0/
> python testing/test.py
Hello from fun 1.0
Hello World!
# Now, I want to use another version of an actor
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.1
> python testing/test.py
Hello from fun 1.1
Hello World! |
This way, it's possible to create custom work environments where various actors can be installed, reinstalled, where we can mix different versions of actors. However, due to pip related limitations we can have only one actor at the time inside virtual environment.
Accessing actors via wrappers
In this scenario, we provide place with all the actors WORKSPACE and via the wrapper we allow user to use different versions of actor. The structure of workspace directory remains the same. We have to make sure we point to this place using environment variable
| Code Block |
|---|
# Make sure that Python based wrapper will be able to load actor's code
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace |
for each actor we need a wrapper
| Code Block |
|---|
'''
actor_demo_loader.py
'''
import os
import sys
import platform
import imp
def import_actor(actor_name='actor_demo', version=None):
workspace = os.getenv('PYTHON_WORKSPACE')
if workspace is None:
raise Exception(
'It looks like PYTHON_WORKSPACE is not set')
python_ver = platform.python_version_tuple()
sys_arch = platform.machine()
actor_path = os.path.join(workspace, os.getenv('IMAS_VERSION'), actor_name, version, actor_name)
print('Importing actor: \n ' + actor_path)
if not os.path.isdir(actor_path):
raise Exception(
'Directory {} does not exists, you have probably not generated any actor'.format(
actor_path))
sys.path.append(actor_path)
fp, pathname, description = imp.find_module('wrapper', [actor_path])
_imas_module = imp.load_module('wrapper', fp, pathname, description)
globals().update(_imas_module.__dict__)
del _imas_module |
With this, we can use actors inside regular Python environment (actors are never installed inside site-packages).
other_actor - this is another actor
| |-- 1.0
| | `-- other_actor
| |-- 1.1
| | `-- other_actor
| `-- 1.2
| `-- other_actor
`-- 3.24.0
|-- actor_demo
| |-- 1.0
| | `-- actor_demo
| |-- 1.1
| | `-- actor_demo
| `-- 1.2
| `-- actor_demo
`-- other_actor
|-- 1.0
| `-- other_actor
|-- 1.1
| `-- other_actor
`-- 1.2
`-- other_actor
|
In this scenario, actor is treated as package. It means, we can either install it using pip command or we can access it via wrapper module by setting some artificial access point for all the actors - PYTHON_WORKSPACE. This approach provides huge flexibility when it comes to distributing actors. It's very easy to create virtual environment and install actors inside it.
Source code - actor
Source code directory contains two sample actors. This is a prototype of what FC2K will generate while building an actor.
| Code Block |
|---|
src/
|-- compile.sh - helper script for building actors (explained later)
|-- actor_demo - actor: actor_demo
| |-- demo.c.template - this is the native code - one that actor provides
| |-- MANIFEST.in - this is the list of files that are part of the Python package (used by pip)
| |-- setup.py.template - this is the description of the package (used by pip)
| `-- wrapper.py - this is the wrapper - it exports Python like interface to native code
`-- other_actor - actor: other actor
|-- demo.c.template
|-- MANIFEST.in
|-- setup.py.template
`-- wrapper.py |
Building the actors
With the structure above, we can easily create actors for different IMAS releases. It will automatically create all the components of the Python packages
| Code Block |
|---|
# This is the place that will be used as a workspace
mkdir -p workspace
# In case we have something inside, we can remove it
rm -rf workspace/*
# We have to make sure that PYTHON_WORKSPACE points to existing location
setenv PYTHON_WORKSPACE `pwd`/workspace
# Let's start with default settings
module purge
module load cineca
module load imasenv
# Now, we can simulate the process of releasing different actors
# with different version numbers
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2
./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd
# Now, let's switch the environment to different version
# of IMAS
module purge
module load cineca
module load imasenv/3.23.2/ual/4.1.5/1.2
# And let's rebuild the actors
pushd src
./compile.sh actor_demo 1.0
./compile.sh actor_demo 1.1
./compile.sh actor_demo 1.2
./compile.sh other_actor 1.0
./compile.sh other_actor 1.1
./compile.sh other_actor 1.2
popd |
once we have completed the build process, the workspace will look like this:
| Code Block |
|---|
workspace/
|-- 3.23.2
| |-- actor_demo
| | |-- 1.0
| | | |-- actor_demo
| | | | |-- __init__.py
| | | | |-- libcore.so
| | | | `-- wrapper.py
| | | |-- MANIFEST.in
| | | `-- setup.py
| | |-- 1.1
| | | |-- actor_demo
| | | | |-- __init__.py
| | | | |-- libcore.so
| | | | `-- wrapper.py
| | | |-- MANIFEST.in
| | | `-- setup.py
| | `-- 1.2
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| `-- other_actor
| |-- 1.0
| | |-- MANIFEST.in
| | |-- other_actor
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | `-- setup.py
| |-- 1.1
| | |-- MANIFEST.in
| | |-- other_actor
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | `-- setup.py
| `-- 1.2
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
`-- 3.24.0
|-- actor_demo
| |-- 1.0
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| |-- 1.1
| | |-- actor_demo
| | | |-- __init__.py
| | | |-- libcore.so
| | | `-- wrapper.py
| | |-- MANIFEST.in
| | `-- setup.py
| `-- 1.2
| |-- actor_demo
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| |-- MANIFEST.in
| `-- setup.py
`-- other_actor
|-- 1.0
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
|-- 1.1
| |-- MANIFEST.in
| |-- other_actor
| | |-- __init__.py
| | |-- libcore.so
| | `-- wrapper.py
| `-- setup.py
`-- 1.2
|-- MANIFEST.in
|-- other_actor
| |-- __init__.py
| |-- libcore.so
| `-- wrapper.py
`-- setup.py |
As you can see, each unique directory pointed by IMAS_VERSION/ACTOR_NAME/ACTOR_VERSION contains self sufficient Python package. This package can be either accessed using helper scripts (Python loaded for the actor) or can be installed using pip.
Virtual environments
Virtual environment provides user based separation of Python environment from the system one. There are multiple choices when it comes to "virtualization" of Python environment:
- https://docs.python.org/3/tutorial/venv.html
- https://docs.conda.io/en/latest/
- https://www.anaconda.com
In this sample, I will use one that is available as Python package.
Installation of actor
In this section, I will install actor (version 1.0) execute it and then, I will upgrade actor to version (1.1) and execute it once again.
During the test, I will use very simple test code
| Code Block |
|---|
'''
We are importing both actors inside env.
'''
from actor_demo.wrapper import actor_demo
from other_actor.wrapper import other_actor
''' Once code is loaded, we can access native functions via Python based wrapper code '''
actor_demo()
other_actor() |
wrapper is the name of the module inside both actors. Each actor exports different Python function: actor_demo in case of actor_demo actor, and other_actor in case of other_actor.
| Code Block |
|---|
# Initialise virtual environment
> virtualenv --no-site-packages -p /gw/switm/python/2.7/bin/python2.7 venv-27
> source venv-27/bin/activate.csh
# Install actors with version 1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.0
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.0
# Now, let's use the actors in the workflow
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.24.0 - 1.0
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.24.0 - 1.0
Hello World!
# If we decide to use different version of the code, it's still possible
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.1
> pip install --upgrade --force-reinstall ./workspace/3.24.0/other_actor/1.1
> python testing/test.py
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.24.0 - 1.1
Hello World!
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.24.0 - 1.1
Hello World! |
This way, it's possible to create custom work environments where various actors can be installed, reinstalled, where we can mix different versions of actors. However, due to pip related limitations we can have only one actor at the time inside virtual environment.
Accessing actors via wrappers
In this scenario, we provide place with all the actors - WORKSPACE - and via the wrappers we allow user to use different versions of actor. The structure of workspace directory remains the same. We have to make sure we point to this place using environment variable
| Code Block |
|---|
# Make sure that Python based wrapper will be able to load actor's code
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace |
for each actor we need a wrapper
| Code Block |
|---|
'''
actor_demo_loader.py
'''
import os
import sys
import platform
import imp
def import_actor(actor_name='actor_demo', version=None):
workspace = os.getenv('PYTHON_WORKSPACE')
if workspace is None:
raise Exception(
'It looks like PYTHON_WORKSPACE is not set')
python_ver = platform.python_version_tuple()
sys_arch = platform.machine()
actor_path = os.path.join(workspace, os.getenv('IMAS_VERSION'), actor_name, version, actor_name)
print('Importing actor: \n ' + actor_path)
if not os.path.isdir(actor_path):
raise Exception(
'Directory {} does not exists, you have probably not generated any actor'.format(
actor_path))
sys.path.append(actor_path)
fp, pathname, description = imp.find_module('wrapper', [actor_path])
_imas_module = imp.load_module('wrapper', fp, pathname, description)
globals().update(_imas_module.__dict__)
del _imas_module |
With this, we can use actors inside regular Python environment (actors are never installed inside site-packages).
| Code Block |
|---|
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace
# I am running workflow with version 1.0 of both actors
#
> python testing/test-wrapper-1.0.py
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/actor_demo/1.0/actor_demo
- I am a native code of the actor. It's name is 'actor_demo'.
Hello from the native function - 'fun': 3.23.2 - 1.0
Hello World!
Importing actor:
/gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.23.2/other_actor/1.0/other_actor
- I am a native code of the actor. It's name is 'other_actor'.
Hello from the native function - 'thecode': 3.23.2 - 1.0
Hello World!
# Now, let's try to run workflow with version 1.1
# |
| Code Block |
> module load imasenv > setenv PYTHON_WORKSPACE `pwd`/workspace > python testing/test-wrapper-1.01.py Importing actor: /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.2423.02/actor_demo/1.01/actor_demo Hello from fun 1.0 Hello World! > python testing/test-wrapper-1.1.py- I am a native code of the actor. It's name is 'actor_demo'. Hello from the native function - 'fun': 3.23.2 - 1.1 Hello World! Importing actor: /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.2423.02/other_actor_demo/1.1/other_actor_demo Hello from fun - I am a native code of the actor. It's name is 'other_actor'. Hello from the native function - 'thecode': 3.23.2 - 1.1 Hello World! |
However, this solution has small issue. It requires explicit selection of the module. It means, it's not quite a Python way of loading modules. Note the difference between test-wrapper-1.0.py
| Code Block |
|---|
import actor_demo_loader import other_actor_demo_loader actor_demo_loader.import_actor(version='1.0') actor_demo_loader.actor_demo() other_actor_loader.import_actor(version='1.0') other_actor_loader.other_actor() |
and test-wrapper-1.1.py
| Code Block |
|---|
import actor_demo_loader import other_actor_loader actor_demo_loader.import_actor(version='1.1') actor_demo_loader.actor_demo() other_actor_loader.import_actor(version='1.1') other_actor_loader.other_actor() |
as you can see, we have to explicitly select version of the actor. An alternative approach would be importing actors by embedding version as a part of package (sort of: from actor_demo.1_0.actor_demo import wrapper). Anyway, there is no simple way of making the very same workflow to be compatible with different versions of actor without altering workflow's code, environment variables, whatever the way of selecting the version we choose.
...