New UAL LowLevel development

1.1. GIT resources 

• Access-layer: branch improvement/develop_new_low_level 
  NOTE: this branch is a clean fork of develop where we imported sources from improvement/new_low_level (which was based on Access Layer Unified, so with a messy history). If you have things in improvement/new_low_level, or any other branch of it, please import them into this new branch by one of the following ways: if changes are in a very small number of commits, you can try to cherry-pick them, otherwise, I would simply advise (only this one time import) to do from improvement/develop/new_low_level a 'git checkout mybranch myfile' (for files which do not exist yet in improvement/new_low_level) or 'git checkout --patch mybranch myfile' (for existing files, allowing you do decide by hunks which part to keep or not).
   
• [Update] Data-dictionary: branch improvement/new_low_level_rebased_3.18.0_structure_gen 
  NOTE: this branch of the DD aims at improving IDSDef.xml so it contains needed info to build all automatically generated source codes, especially when it comes to sub-structure definition. It is also subject of some re-basing to keep it up to date with the last tagged version. This is subject to some changes as we discuss fixes and cleaning with Frederic. History of versions previously in usage with the new AL is as follow:
  • improvement/new_low_level_rebased_3.16.0_structure_gen
  • improvement/new_low_level_rebased_3.16.0
  • improvement/all_in_utilities
  • improvement/new_lowlevel
    
    

1.2. Build notes

This version should be already installer compatible (see note below), but as the installer does not provide the best environment to develop/test the UAL, we are bypassing it and using our own script to set a compatible compilation environment. The core of the new lowlevel layer is developed in C++ and requires a compiler implementing some of the C++11 standards (Intel >= 13, and GCC >= 4.9). You will note that currently IFORT is disable from Makefile.common, as this version of the DD crashes ifort (typically when compiling edge_transport_put/get.f90): we hope to avoid this issue as soon as get/put modules have been implemented using subroutines at the sub-structure level (so far done only for copy_struct and deallocate_struct, will still require some update to IDSDef.xml, see DD note above).

1.2.1.1.1. Preamble

You need to build the data-dictionary in order to create the IDSDef.xml, for that the only requirement is to load the saxon module (called "saxon-HE" at ITER, and "saxon" at GW), then:

cd data-dictionary
make 

As we might not use the installer, we need to make a link in the access-layer root dir which is named 'xml' and points out to where your data-dictionary clone is. This of course needs to be done only once. For instance, if you have cloned both repos from the same place:

cd access-layer
ln -s ../data-dictionary xml 

1.2.1.1.2. Setup on Gateway

To ease this and make sure we ended up with a identical environment/build when debugging the backend with Gabriele, I've added 2 scripts (bash or tcsh) to be sourced before compiling the UAL or running any tests. They can be found at the root of the access-layer, and are sufficient regardless of your initial environment (they take care or purging and loading the correct modules):

source setup.GW.csh

or

source setup.GW.bash

1.2.1.1.3. Setup on ITER cluster

We were doing most of the development and testing work on the Gateway, but when it is down or just to make sure it still compiles/runs the same on the ITER cluster you, I just added equivalent script setting the environment at ITER:

source setup.ITER.bash

1.2.1.1.4. Compilation

To build the lowlevel+backends only:

cd lowlevel
make install

To build everything (with Fortran HLI, but try to be reasonable with the number of parallel threads, even if I usually don't ):

make -j16 install 

1.2.1.1.5. Through the installer 

Set the following environment before calling make (please adapt IMAS_HOME to your local need):

export IMAS_HOME=/home/ITER/hoeneno/test-install
export IMAS_VERSION=improvement/new_low_level_rebased_3.18.0_structure_gen
export UAL_VERSION=improvement/develop_new_low_level
export IMAS_MAJOR=3

export IMAS_CPP=no
export IMAS_PYTHON=no
export IMAS_JAVA=no
export IMAS_MATLAB=no
export IMAS_G95=no
export IMAS_IFORT=no
export IMAS_PYTHON=no
export IMAS_PYTHON2=no
export IMAS_PYTHON3=no
export IMAS_JAVA=no

1.2.1.1.6. Building the tests

We have hand-made Fortran test programs 

cd temporary_tests
make -j 

and the auto generated testSuite

cd fortraninterface/tests/generator
make 

1.3. Sources organization (what-where)

1.3.1. access-layer/lowlevel

It contains the C++ implementation of the new lowlevel, following the naming convention ual_"somename" (no special characters in "somename"). It is composed of a hierarchy of Context classes (in ual_context.h/cpp), a hierarchy of exception classes (in ual_exception), some constants definitions (ual_const and ual_defs), the backend abstract class (ual_backend) and some of its implementations (dummy no_backend, mdsplus_backend, memory_backend), and finally the core of the lowlevel itself with C bindings (ual_lowlevel).

Other source files are left-overs from the old lowlevel and contains (for reference only):

• the old version of the so called "compatibility layers" (which means the wrapper in F77, JNI and others around C functions.
  These are supposed to be simple direct translation, no logic, but unfortunately in the past it was not the case... if such compatibility layer is still required for a given HLI, we propose to keep it in the HLI directory (like it was already done in the past with Python, and as it is done with Fortran now) and carefully control that nothing more than translation is done.
• the old implementation of some additional services like catalog, remote, logger... 
• the old implementation of backends for mdsplus and hdf5

The only exception from these left overs is ual_low_level.h/c, which corresponds to the old LL API, but now implemented using calls to the new LL API. Thus it is a possibility to develop the HLI from this almost old-API based version (as it was done, for historical reason in Fortran), but it is not a requirement! Plus the signature of some of this old-API has to be slighly changed in order to reflect the most logically different aspects, especially concerning context management and array of structure operations.

1.3.1.1.1. Approach and few notable changes

The main idea is to have single read and single write function passing the data using void pointer with extended set of arguments allowing to specify type, dimension and size. Then the type of read/write operation (timed, sliced, which IDS, etc...) is handled by a reduced set of functions changing a "context". All new API functions starts with "ual_" and are of two types:

1. ual_begin_*, ual_end_action, ual_iterate_over_arraystruct have effects on the context: they can change it (taking one type of context as input and providing a different type as output) or modify a given context (ual_iterate_over_arraystruct), the scope of their action is the lowlevel only
2. ual_read_data, ual_write_data and ual_delete_data are the I/O operations that are forwarded to the backend
3. ual_open_pulse, ual_close_pulse, 

Another important change is the disappearance of "string" concept from the new LL, as string is not a data but has different meaning depending of the programming language. Instead in the lowlevel we only deal with characters, a string being now a 1D array of chars.

1.3.2. access-layer/fortraninterface

Not much there as sources are to be mostly generated from the IDSDef.xml. IDSDef2F90Routines.xsl is the main XSL transform there, ones based on xsd are (xsd2copy_structures, xsd2deallocate_structures) or will be (xsd2F90TypeDef) deprecated when IDSDef.xml will be complete. Note that we are already in this branch using the single Makefile (merged from develop) instead of XSL generated makefile.

The Fortran compatibility layer is under "wrapper" subdir. It defines using ISO_C_BINDING all lowlevel functions (old and new APIs) and implements wrappers (Fortran subroutines) around these functions. It also defines Fortran version of the constants (if someone see a cleaner and extensible way to define uniquely these constants I would be happy to change the current implementation).

1.4. New lowlevel developer documentation 

When you will compile the lowlevel, it will also build Doxygen documentation of the related classes and sources. There might be some small descrepencies but it should mostly be up to date and give a usefull overview of the APIs and meaning of the arguments. By browsing per files, you can check the C-API of the new LL in 'ual_lowlevel.h' with complete description of the arguments and some example usage (in implementing the old LL-API).

Some generic rules that drives the current approach:

1. All lowlevel operations require a Context (except creation of PulseContext, which is the most basic one)
2. Context hierarchy is: PulseContext < OperationContext < ArraystructContext
3. Creating a new type of Context (subclass) always requires passing the superclass: Arraystruct context are obtained from OperationContext or ArraystructContext, OperationContext are obtained from PulseContext
4. Each Context has global scope (a table of all open Context is maintained at the lowlevel/C++) and an extent which starts at is creation (from a ual_begin_* method) to its destruction (from ual_end_action)
5. Returned Context ID are returned with negative value when an error has occurred 
6. Time bases 
   1. can be relative or absolute paths
      1. absolute paths (timebase string starts with "/", typically "/time") refer to the root of the IDS
      2. relative paths (not starting with "/", could start with "name/" or even "../") refer to the current context (if it is an operationContext then the path is relative to the IDS root, if it is an ArraystructContext then the path is relative to this AoS)
   2. are set if the data (field or AoS) is dynamic
      1. it refers to root "IDS/time" if the IDS is in homogeneous time base mode (IDS/ids_properties/homogeneous_time=1) 
      2. it refers to field specified in attribute "coordinate_AosParent_relative" in the IDSDef.xml if the IDS is in heterogeneous time base mode (IDS/ids_properties/homogeneous_time=0)
   3. are not set if the data is static or constant
   4. are not set if the data is dynamic but contained in a dynamic AoS
7. There can be only one dynamic AoS within an ancestry line

Hereafter are some examples of typical lowlevel chain of calls for some generic operations:

Working with a pulsefile:

pulsectx = ual_begin_pulse_action(backendID, shot, run, user, tokamak, version);
status = ual_open_pulse(pulsectx, mode, options);
...
status = ual_close_pulse(pulsectx, mode, options);   /* ual_end_action(pulsectx) is called in ual_close_pulse, only exception where not called explicitly to end Context lifespan */

Getting an IDS:

opctx = ual_begin_global_action(pulsectx, idsname, READ_OP);
if (opctx < 0)  stop_on_error
...
/* getting fields and AoS */
status = ual_read(opctx, fieldpath, timebasepath, data, datatype, dim, size)
/* see further below for AoS */
...
status = ual_end_action(opctx);

Getting a slice of IDS:

opctx = ual_begin_slice_action(pulsectx, idsname, READ_OP, time, interpmode);
if (opctx < 0)  stop_on_error
...
/* getting fields and AoS: exact same sequence of calls that in "Getting an IDS" case */ 
...
status = ual_end_action(opctx);

Getting an AoS:

aos1ctx = ual_begin_arraystruct_action(opctx, aos1path, aos1timebase, &aos1size);
if (aos1ctx < 0)  stop_on_error
if (aos1size > 0)  allocate(aos1, aos1size)
for (i1=0; i1 < aos1size; i1++)
{
   ...
   /* getting fields within an element of the AoS */
   status = ual_read_data(aos1ctx, fieldpath, timebasepath, data, datatype, dim, size); 
   ...
   /* getting a nested AoS */
   aos2ctx = ual_begin_arraystruct_action(aos1ctx, aos2path, aos2timebase, &aos2size);
   if (aos2ctx < 0)  stop_on_error
   if (aos2size > 0)  allocate(aos2, aos2size)
   for (i2=0; i2 < aos2size; i2++) 
   {
      status = ual_read_data(aos2ctx, fieldpath, timebasepath, data, datatype, dim, size);
      ...
      status = ual_iterate_over_arraystruct(aosctx2, 1); // second argument is step size
   }
   status = ual_end_action(aos2ctx);
   ...
   status = ual_iterate_over_arraystruct(aosctx1, 1);
}
status = ual_end_action(aos1ctx);

Deleting an IDS:

opctx = ual_begin_global_action(pulsectx, idsname, WRITE_OP);
if (opctx < 0)  stop_on_error
...
status = ual_delete_data(opctx, fieldpath);
...
status = ual_end_action(opctx);

Putting an IDS:

/* first do remove IDS from pulsefile, calling "Deleting an IDS" highlevel function */
/* then check if IDS.homogeneous_time is set, and if set to 1, check if IDS.time is allocated */
opctx = ual_begin_global_action(pulsectx, idsname, WRITE_OP);
if (opctx < 0)  stop_on_error
...
/* putting fields and AoS, whole size */
/* only if scalar/vector field is set/allocated do */
status = ual_write(opctx, fieldpath, timebasepath, data, datatype, dim, size)
/* see further below for AoS */
...
status = ual_end_action(opctx);

Putting a slice of IDS:

opctx = ual_begin_slice_action(pulsectx, idsname, WRITE_OP, time, UNDEFINED_INTERP);
if (opctx < 0)  stop_or_error
...
/* putting fields and AoS, only timed fields, with last dim (time) with size=1, e.g. 4D data will be put as 4D data with size=[x,y,z,1] */
...
status = ual_end_action(opctx);

Putting an AoS:

aos1ctx = ual_begin_arraystruct_action(opctx, aos1path, aos1timebasepath, &aos1size);
if (aosctx1 < 0)  stop_on_error
for (i1=0; i1 < aos1size; i1++)
{
   ...
   /* putting fields within an element of the AoS */
   status = ual_write_data(aos1ctx, fieldpath, timebaspath, data, datatype, dim, size); 
   ...
   /* putting a nested AoS */
   aos2ctx = ual_begin_arraystruct_action(aos1ctx, aos2path, aos2timebasepath, &aos2size);
   if (aos2ctx < 0)  stop_on_error
   for (i2=0; i2 < aos2size; i2++)
   {
      status = ual_write_data(aos2ctx, fieldpath, timebasepath, data, datatype, dim, size);
      ...
      status = ual_iterate_over_arraystruct(aosctx2, 1); // second argument is the step size
   }
   status = ual_end_action(aos2ctx);
   ...
   status = ual_iterate_over_arraystruct(aosctx1, 1);
}
status = ual_end_action(aos1ctx);