Current solution

It looks like (from user's perspective) we miss simplified way of choosing "dressed" Kepler and workflow that will be used to launch ETS. In fact, the same sort of issues will affect Python based workflows.

At the moment, release of Kepler follows (heavy simplified process):

• release of new FC2K (e.g.: version R4.2.6 this will be transparent for users - they don't need FC2K for day to day usage)
• release of new Kepler  (e.g. version R5.0.16_v5.2.2)
• release of all actors (based on tag, e.g.: v5.2.2)
• release of supplementary components (e.g.: kplots - ETS_4.10b.10_v5.6.0/kplots)
• release of workflow (e.g.: ETS_v5.2.2.xml)
• release of autoGui (e.g.: 1.1)

all these components make user based environment.

Main goal

To provide to the user a mechanism that:

• allows to run ETS workflow via ETS GUI
• automatically and transparently updates all ETS components

Assumptions

• User sets up and runs ETS via ETS GUI only 

Module ets 

Module ets configures user working environment by loading consistent set of libraries that were used to build particular version of ETS.

It internally

• loads module itmenv 
• sets particular release of ETS actors 
• sets particular release of ETS workflow (ETS.xml)

Proposed syntax: {{ets/dd-ver/ets-ver e.g. ets/3.27.0/6.3

Script launch_ets.sh (please name it simply "ETS.sh")

This script

• checks if new versions of ETS environment are available
• in case a new version was released, allows the user to decide is local installation should be updated
• if YES - automatically updates local user installation of ETS environment 

Use cases:

Use-case 1 - Running existing installation of ETS

1) Precondition - first run or no ETS versions released since last run

2) Setting ETS environment and launching ETS

# 1. log in to machine
# 2. initialise environment

> module load ets

# 3. run the autoGui

> launch_ets.sh

autoGui is started, user can use his version of ETS with the default workflow loaded by environment.

Use-case 2 - Running already installed version of ETS (even if new version was released)

1) Precondition - ETS environment updated

In this scenario, system was upgraded and new release of tools was installed

• release of new Kepler  (e.g. version R5.0.16_v5.2.3)
• release of all actors (based on tag, e.g.: v5.2.3)
• release of supplementary components (e.g.: kplots)
• release of workflow (e.g.: ETS_v5.2.3.xml)

2) Setting ETS environment and launching ETS

# 1. log in to machine
# 2. initialise environment

> module load ets

# 3. run the autoGui

>  launch_ets.sh
There is a new release of ETS. Do you want to upgrade [Y/N]?: N

User presses N and currently installed version of autoGui [is autoGui being "installed" locally?] and current version of environment (Kepler, set of ETS actors, and workflow file) is used. No changes inside user's settings are done.

Use-case 3 - Running ETS  (automatic update of ETS installation)

1) Precondition - ETS environment updated

In this scenario, system was upgraded and new release of tools was installed

• release of new Kepler  (e.g. version R5.0.16_v5.2.3)
• release of all actors (based on tag, e.g.: v5.2.3)
• release of supplementary components (e.g.: kplots)
• release of workflow (e.g.: ETS_v5.2.3.xml)

2) Setting ETS environment and launching ETS

# 1. log in to machine
# 2. initialise environment

> module load ets

# 3. run the autoGui

>  launch_ets.sh
There is a new release of ETS. Do you want to upgrade [Y/N]?: Y

User presses Y. Script performs upgrade of all tools installed for the user and starts most recent version of autoGui and most recent version of Kepler. Previous version of user's installation are preserved and it will be possible to go back to this version later.

Preliminary investigation

All the above solutions are keeping the current approach which requires user to have a sort of local install of Kepler (composed of links or of the full sources).

Before going in that direction, we want to check the technical feasibility of running a workflow with a Kepler that remains entirely in public place. 

Kepler 

We need to check that running Kepler in such mode is possible, it involves a few actions, mainly on Michal, but with help of others when/where needed:

Preparation for the test

To simulate the case where we run everything from the central installation I am working on a local copy with all the access rights set to read only

> chmod a-w kepler

Test installation is here: /gss_efgw_work/work/g2michal/cpt/development/isolated_kepler/kepler 

Running from public installation

First test is related to running Kepler interactively from public install (can Kepler log be stored elsewhere) - I am using copy of: ETS_4.10b.10_v5.2.2 

We have first issue related to local changes inside KEPLER directory

> ant run
...
...
  [compile] Compiling itm...

BUILD FAILED
/gss_efgw_work/work/g2michal/cpt/development/isolated_kepler/kepler/build-area/build.xml:56: java.io.FileNotFoundException: ./depcache/itm/dependencies.txt (Permission denied)

The above issue can be solved by running from the place where uses has access rights. For example, running it inside directory where user has write access will result in creating directory deepcache, and Kepler will start

> pwd
/gss_efgw_work/work/g2michal/cpt/development/isolated_kepler
> ant -f ant -f kepler/build-area/build.xml run
> tree depcache
depcache
|-- actors
|   `-- dependencies.txt
|-- authentication
|   `-- dependencies.txt
|-- authentication-gui
|   `-- dependencies.txt
|-- component-library
|   `-- dependencies.txt
|-- configuration-manager
|   `-- dependencies.txt
|-- core
|   `-- dependencies.txt
|-- data-handling
|   `-- dependencies.txt
|-- dataone
|   `-- dependencies.txt
|-- dataturbine
|   `-- dependencies.txt
|-- display-redirect
|   `-- dependencies.txt
|-- ecogrid
|   `-- dependencies.txt
|-- event-state
|   `-- dependencies.txt
|-- gui
|   `-- dependencies.txt
|-- io
|   `-- dependencies.txt
|-- itm
|   `-- dependencies.txt
|-- job
|   `-- dependencies.txt
|-- loader
|   `-- dependencies.txt
|-- module-manager
|   `-- dependencies.txt
|-- module-manager-gui
|   `-- dependencies.txt
|-- opendap
|   `-- dependencies.txt
|-- r
|   `-- dependencies.txt
|-- repository
|   `-- dependencies.txt
|-- sms
|   `-- dependencies.txt
|-- ssh
|   `-- dependencies.txt
`-- util
    `-- dependencies.txt

There is, an issue with Log4J based logger

[compile] Compiling itm...

run:
      [run] JVM Memory: min = 1g,  max = 4g, stack = 20m
      [run] log4j.properties found in CLASSPATH: /gss_efgw_work/work/g2michal/cpt/development/isolated_kepler/kepler/itm/resources/log4j.properties
      [run] log4j:ERROR setFile(null,true) call failed.
      [run] java.io.FileNotFoundException: keplerLog4J.log (Permission denied)

FIXED: I was able to overcome this issue (and at the same time changing work dir to different location) by implementing something like this inside file: kepler/build-area/src/org/kepler/build/Run.java 

Java java = new Java();
java.setDir(new java.io.File(System.getenv("KEPLER_WORK_DIR")));

Kepler is started in two phases. First one, responsible for collecting modules and starting another one (workflow execution). This process is maintained by Ant. It is possible to overcome defaults and run the code in any directory (e.g. current one). At the moment, solution is little bit ugly (I am reading location - where Kepler is supposed to be started - from environment).

Running FC2K based actor inside Kepler

Running Kepler interactively with FC2K actors inside the workflow. We have to make sure that current directory can be set to any location outside of the Kepler. We run simple cases, then ETS-like ones, where some might read/write files in current directory

In order to run test case for ETS make sure to follow these steps:

# prepare env.

> module load itmenv/ETS_4.10b.10_v5.6.0
> mkdir $ITMWORK/isolated_kepler_test
> cd $ITMWORK/isolated_kepler_test
> source $ITMSCRIPTDIR/ITMv2 test
> setenv KEPLER /gss_efgw_work/work/g2michal/cpt/development/isolated_kepler/ETS_4.10b.10_v5.6.1/kepler
> svn co https://gforge6.eufus.eu/svn/keplerworkflows/tags/ETS_4.10b.10_v5.6.0

# do not go inside $KEPLER!
> $KEPLER/kepler.sh ETS_4.10b.10_v5.6.0/ETS_WORKFLOW.xml

# execute the workflow

you should be able to execute the whole workflow and see the results as below

In case actors create some files (e.g. for internal processing or as intermediate results), these files are no longer created inside $KEPLER. Instead, these are created in current directory - place where you have started Kepler

> /gss_efgw_work/work/g2michal/cpt/development/isolated_kepler
> tree --charset=ascii -L 1
.
|-- EQDSK_COCOS_02_POS.OUT
|-- EQDSK_COCOS_13.OUT
|-- ETS_4.10b.10_v5.6.0
|-- ETS_4.10b.10_v5.6.1
|-- EXPEQ_KEPLER.IN
|-- EXPEQ.OUT
|-- EXPEQ.OUT.TOR
|-- EXPTNZ.OUT
|-- kepler
|-- kepler.log
|-- keplerLog4J.log
`-- NOUT

kepler.log file - one that contains log of Kepler's execution is also created in current directory.

No special actions are needed from user's side. Everything is set up inside kepler.sh script.

Standalone mode

Running these actors in standalone mode (checking the behaviour of the copy of sources/exec from initial place in Kepler to KEPLEREXECUTION/sandbox)

Debug mode

Running these actors in debug mode (checking the behaviour of the copy of sources/exec from initial place in Kepler to KEPLEREXECUTION/sandbox)

Batch mode

Running these actors in batch mode (checking the behaviour of the copy of sources/exec from initial place in Kepler to KEPLEREXECUTION/sandbox)

SBK based execution

Checking sbk execution - with simple workflow and then with ETS workflow

Kplots and other scripts 

We need to check that these scripts (currently copied within Kepler sources when Kepler is being dressed-up) can be stored outside Kepler (public place, versioned, and made available through module) and referenced from the workflow: kplots module will set up a KPLOTS_HOME which points to dir containing all the scripts, we need to check that reading this variable from the workflow will work correctly (getenv). Action on Olivier to create public install of kplots with modules, and on Dmitriy to update ETS workflow to link to Kplots scripts using the env variable set by module. These actions do no depend on Kepler actions above.