Page History

...

In this scenario I assume complete separation of Kepler and Python. FC2K should allow choosing whether we want to generate actor for given workflow platform: Python, Kepler, something else in the future. Once platform is selected, FC2K generates source code and "Weighted Companion Cube" - one that contains all the native codes, libraries, etc. - for given platform.

Kepler

TODO: we have to decide whether we want to update the way of generation for Kepler or notDue to the fact that Kepler will be used for ETS only (probably) it seems like there is no much sense in redesigning the whole idea of actor generation.

Python

I suggest moving towards Python packages that are self contained packages consisting of:

• python wrapper
• shared libraries
• standalone binary (MPI based execution)
• default parameters

Each package should be either installed inside user's virtual environment or should be available via wrapper module - this way, it will be possible to provide users with any version of actor and Python workflow will be composed of loosely coupled actors.

...

Inside, we have a sample structure of Python module with native code: demo.c.

src/
|-- actor_demo
|.
|-- src                         - source code of the actor (native code)
|-- testing                     - scripts used for testing the solution
|-- venv-27                     - virtual environment
`-- workspace                   - workspace (as described by Bartek)
    `-- 3.24.0
        `-- actor_demo          - actor directory (this is not Python package)
            |-- __init__.py
1.0             - version of the actor
            |   `-- sample.py
| actor_demo  - this is Python package (that contains module)
            `-- demo1.c
1
                `-- setup.pyactor_demo

In this scenario, actor is treated as package. It means, we can either install it using pip command or we can access it via wrapper module by setting some artificial access point for all the actors - PYTHON_WORKSPACE. This approach provides huge flexibility when it comes to distributing actors. It's very easy to create virtual environment and install actor inside it.

Virtual environments

Virtual environment provides user based separation of Python environment from the system one. There are multiple choices when it comes to "virtualization" of Python environment:

• https://docs.python.org/3/tutorial/venv.html
• https://docs.conda.io/en/latest/
• https://www.anaconda.com

In this sample, I will use one that is available as Python package.

Installation of actor

In this section, I will install actor (version 1.0) execute it and then, I will upgrade actor to version (1.1) and execute it once again.

During the test, I will use very simple test code

from actor_demo import wrapper

wrapper.actor_demo()

wrapper is the name of the file inside actor_demo module - it's an arbitrary choice.

# Initialise virtual environment 
> virtualenv --no-site-packages -p /usr/gw/switm/python/2.7/bin/python2.7 samplevenv-27
> source samplevenv-27/bin/activate.csh

# Install actor with version 1.0
> pip install --upgrade --force-reinstall ./src//workspace/3.24.0/actor_demo/1.0/
> python testing/test.py
Hello from fun 1.0
Hello World!

# Now, I want to use another version of an actor
> pip install --upgrade --force-reinstall ./workspace/3.24.0/actor_demo/1.1
> python testing/test.py
Hello from fun 1.1
Hello World!

This way, it's possible to create custom work environments where various actors can be installed, reinstalled, where we can mix different versions of actors. However, due to pip related limitations we can have only one actor at the time inside virtual environment.

Accessing actors via wrappers

In this scenario, we provide place with all the actors  WORKSPACE and via the wrapper we allow user to use different versions of actor. The structure of workspace directory remains the same. We have to make sure we point to this place using environment variable

# Make sure that Python based wrapper will be able to load actor's code
> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace

for each actor we need a wrapper

'''
actor_demo_loader.py
'''

import os
import sys
import platform
import imp

def import_actor(actor_name='actor_demo', version=None):

  workspace = os.getenv('PYTHON_WORKSPACE')

  if workspace is None:
    raise Exception(
      'It looks like PYTHON_WORKSPACE is not set')

  python_ver = platform.python_version_tuple()
  sys_arch = platform.machine()

  actor_path = os.path.join(workspace, os.getenv('IMAS_VERSION'), actor_name, version, actor_name)
  print('Importing actor: \n ' + actor_path)

  if not os.path.isdir(actor_path):
    raise Exception(
      'Directory {} does not exists, you have probably not generated any actor'.format(
        actor_path))

  sys.path.append(actor_path)
  fp, pathname, description = imp.find_module('wrapper', [actor_path])
  _imas_module = imp.load_module('wrapper', fp, pathname, description)

  globals().update(_imas_module.__dict__)
  del _imas_module

With this, we can use actors inside regular Python environment (actors are never installed inside site-packages).

> module load imasenv
> setenv PYTHON_WORKSPACE `pwd`/workspace
> python testing/test-wrapper-1.0.py
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.24.0/actor_demo/1.0/actor_demo
Hello from fun 1.0
Hello World!
> python testing/test-wrapper-1.1.py
Importing actor:
 /gss_efgw_work/work/g2michal/cpt/development/python_modules/simple/workspace/3.24.0/actor_demo/1.1/actor_demo
Hello from fun 1.1
Hello World!

However, this solution has small issue. It requires explicit selection of the module. It means, it's not quite a Python way of loading modules. Note the difference between test-wrapper-1.0.py 

import actor_demo_loader

actor_demo_loader.import_actor(version='1.0')
actor_demo_loader.actor_demo()

and test-wrapper-1.1.py 

import actor_demo_loader

actor_demo_loader.import_actor(version='1.1')
actor_demo_loader.actor_demo()

as you can see, we have to explicitly select version of the actor. An alternative approach would be importing actors by embedding version as a part of package (sort of: from actor_demo.1_0.actor_demo import wrapper). Anyway, there is no simple way of making the very same workflow to be compatible with different versions of actor without altering workflow's code, environment variables, whatever the way of selecting the version we choose. 

One common installation of actor resources

...