Versions Compared

Old Version 6

changes.mady.by.user Unknown User (olivier.hoenen@ipp.mpg.de)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (olivier.hoenen@ipp.mpg.de)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
This page describes the important API differences of the IMAS Access Layer version 4 (AL4), especially for users which have already experience with previous version 3 (AL3). 
This page is intended as a quick HOWTO, for more details on the AL4 usage please refer to the "AL user guide" (todo: link the user guide here when available from imasfrom ITER's page: https://user.iter.org/?uid=YSQENW&action=get_document).
 

 Info
titleData Dictionary dependency
IMAS modules and versions are composed of both a Data Dictionary version (DD) and the Access Layer version (AL). Access Layer 4.0.0 was released the 29th of March 2019 for the 3.21.1 DD version. While in principle older DD version can be supported, for practical reasons no older DD are planned to be released with AL4. DD 3.21.0 is the last version supported by AL3, all more recent versions can only be released with AL4.
 

Below you will a section on main difference in high-level interface API. Then users API for each high-level language will be compared side by side comparison for AL3 and AL4. todo: add section on backends
 
Main discrepancies
 Info
Names of functions given hereafter might differ slightly in the language specific high-level interfaces, please refer to language specific sections below or to "Access Layer user's guide" for exact names and signatures, as well as for differences which are specific to a given language interface.
  • Reference to a data-entry now requires always to specify user, machine and version, as there is no proper default mechanism. This means that open(),create() functions are now deprecated, and only open_env() and create_env() remain in AL4.
  • Side effect of the change above, calling imasdb is not mandatory anymore (it was with AL3 for using either open(),create() or Kepler), but the script is still present in order to help users to create the local database layout (one time call per machine/version).
  • put_non_timed() function is now deprecated, as this feature was not used when not followed by at least one put_slice(). AL3 chain of put_non_timed()+put_slice() should be  is now translated into a single call to put() where IDS contains only one slice of timed data. Note that this combined , further slices can then be appended through calls to put_slice(). Note that this combined approach is valid in both AL3 and AL4, and that put_slice() has a well defined behavior only when the passed IDS contains one single slice of timed data. Note also that a put() is required before any put_slice() (as only the put stores static data, and static field ids_properties.homogeneous_time is mandatory for obtaining a valid IDS)
  • The caching mechanism, as well as the flush(),discard() operations are deprecated. A new memory backend is introduced in AL4 in order to reduce IO costs of disk accesses when not necessary.
  • New API to select any available backend will be added soon, in the meantime it is possible to use the lowlevel API (if accessible from targeted language interface) to perform this selection (currently available backends are MDSplus, Memory and UDA). 
 
  • , refer to User Guide section 5.14 for more information). 


 Tip
titleMDSplus pulsefiles conversion
The MDSplus models (structure of the pulsefiles) have changed a lot between AL3 and AL4. Reading with AL4 a pulsefile written with AL3 or the other way around is not possible. Nevertheless a conversion script is provided with any AL4 release, so users can attempt to convert an AL3 generated pulse into a AL4 compatible one. This script is developed purely in MDSplus and will work only for pulsefiles with IDS in homogeneous_time and which are well formed. Usage, after having loaded one AL4 IMAS module (or imasenv <>= 3.21.1) is as the following:
 Code Block
ualport.py <shot> <run> <oldDir> <newDir>
	<shot>   : shot number, or 'all' for converting all shots presents in <oldDir>
	<run>    : run number, or 'all' for converting all runs presents in <oldDir>
	<oldDir> : directory containing the source (AL3) pulsefile
	<newDir> : directory where will be stored the destination (converted to AL4) pulsefile
For sake of clarity, it is strongly advised that you use two separate directories (machine names for instance) to differenciate AL3 and AL4 pulses. For instance you can rename your test tokamak (or any other machine name) to test-AL3 and recreate an empty test where AL4 generated or converted pulsefiles will be stored:
 Code Block
mv ~/public/imasdb/test ~/public/imasdb/test-AL3
imasdb test
ualport.py 123 1 ~/public/imasdb/test-AL3/3/0 ~/public/imasdb/test/3/0
 
 
 
n.b. "3" in the oldDir and newDir path of the example above are not referring to AL3! This stands for the major version of the DD (which is identical at the moment).
For converting pulses with run number > 9999 --are you sure you have (important) data there?-- please contact CPT (or use wildcard with correct directory if you understand the layout well enough)



Fortran

  • bindings to lowlevel C/C++ is done explicitly through iso_c_bindings, without reallocating arrays when possible (for all non-character based quantities)

    Warning
    Because of arrays being allocated in C when IDS is obtained from AL (get/get_slice), deallocation / allocation should be done with extra care especially for routines using IDS as inout arguments! A new substructure level deallocation routine was introduced (see table below) but it needs additional logical input specifying if arrays of the substructure were allocated in C (AL) or Fortran (user's code). Two options:
    a) the IDS is deallocate within the user's code scope, then a mix of C/F allocation need to be deallocated accordingly
    b) the IDS is potentially used/extended/deallocated by another code, or is coming from different contexts (AL or another Fortran code), then the safer approach is to work on a copy of the

Fortran

  • bindings to lowlevel C/C++ is done explicitly through iso_c_bindings, without reallocating arrays when possible (for all non-character based quantities)

    Warning
    Because of arrays being allocated in C when IDS is obtained from AL (get/get_slice), deallocation should be done with extra care especially for routines using IDS as inout arguments! A new substructure level deallocation routine was introduced (see table below) but it needs additional logical input specifying if arrays of the substructure were allocated in C (AL) or Fortran (user's code). Two options:
    a) the IDS is deallocate within the user's code scope, then a mix of C/F allocation need to be deallocated accordingly
    b) the IDS is potentially used/extended/deallocated by another code, or is coming from different contexts (AL or another Fortran code), then the safer approach is to work on a copy of the IDS (potentially only for inout IDS which are coming from AL, which can be checked using the new is_c_data routine presented in table below)


  • wrappers around lowlevel API are provided, allowing for instance to select a different backend

 

  • AL 4.1.0 introduces an optional output parameter to main part of the API in Fortran, to return error status that can be then checked in user's code. Added optional parameter is highlighted in green  in the table below)


OperationAL3
OperationAL3
AL4Remarks

creating a data-entry

(MDSplus backend)

call imas_create(name,shot,run,shotref,runref,idx)
call imas_create_env(name,shot,run,shotref,runref,& 
                     idx,user,machine,version)

 

 



call

imas_create_env(name,shot,run,shotref,runref,&

idx,user,

idx,user,machine,version,&
retstatus)

Concept of default machine is deprecated

Note

Arguments name,refshot,refrun can have any (or empty) value, and are kept only for compatibility. A new unique data-entry selection routine will be added to the API, making this routine deprecated at term.

integer(ids_int),intent(out),optional :: retstatus has value 0 upon success

opening a data-entry

(MDSplus backend)

call imas_open(name,shot,run,idx)
call imas_open_env(name,shot,run,idx,&
                   user,machine,version)
 

call

imas_open_env(name,shot,run,idx,&

user,machine,version

user,machine,version,&
retstatus)

Concept of default machine is deprecated

Note

Argument name can have any (or empty) value, and is kept only for compatibility. A new unique data-entry selection routine will be added to the API, making this routine deprecated at term.

integer(ids_int),intent(out),optional :: retstatus has value 0 upon success

writing the complete IDS
call ids_put(idx,idsname,ids)

call

ids_put(idx,idsname,ids,retstatus)

 writing a single slice of an IDS
integer(ids_int),intent(out),optional :: retstatus has value 0 upon success
writing a single slice of an IDS
call ids_put_
call ids_put_
slice(idx,idsname,ids)

call

ids_put_slice(idx,idsname,ids,retstatus)

 
integer(ids_int),intent(out),optional :: retstatus has value 0 upon success
writing only non-timed data of an IDS
call ids_put_non_timed(idx,idsname,ids)
 
DeprecatedDeprecated, replaced by ids_put() on IDS containing only first slice data
reading the complete IDS
call ids_get(idx,idsname,ids)

call

ids_get(idx,idsname,ids,retstatus)

 

reading a single slice of an IDS
call ids_get_slice(idx,idsname,ids,time,interpol)

call

ids_get_slice(idx,idsname,ids,time,interpol,&
retstatus)

Paremeters are defined in ual_defs module for the different interpolation modes (to be used for interpol argument): CLOSEST_INTERP,PREVIOUS_INTERP,LINEAR_INTERP
integer(ids_int),intent(out),optional :: retstatus has value 0 upon success
copying an IDS or one of its substructure
call ids_copy(src,dest)
call ids_copy(src,dest)
 



deallocating an IDS or one if its substructure
call ids_deallocate(ids)
call ids_deallocate(ids)
call ids_deallocate_struct(struct,c_data)
When deallocating a substructure of an IDS, one need to pass a logical argument indicating whether this structure has been obtained allocated in C (obtained from AL get(),get_slice()) or not. This check is performed automatically while deallocating an entire IDS
checking if arrays of an IDS were allocated in C (by AL)
 

call is_c_data(ids,c_data)
Returns the c_data logical which is .true. if the IDS was obtained through the Access Layer, .false. otherwise.
deleting an IDS from a data-entry
call ids_delete(idx,idsname,ids)
call ids_delete(idx,idsname,ids)
 

closing a data-entry
call imas_close(idx)
call 
 
 imas_close(idx,retstatus)
 
 
 
 
C++
 
 
Python
 
 
Matlab
 
 
Java
 
 
integer(ids_int),intent(out),optional :: retstatus has value 0 upon success
enabling/disabling memory-cachecall imas_enable_mem_cache(idx)
call imas_disable_mem_cache(idx)		DeprecatedDeprecated, for in-memory operations use the MEMORY_BACKEND (see User Guide section 5.14)
flushing data from cachecall imas_flush_all(idx)
call imas_flush(idx,idsname)		DeprecatedDeprecated, for in-memory operations use the MEMORY_BACKEND (see User Guide section 5.14)
discarding data from cache
call imas_discard_all(idx)
call imas_discard(idx,idsname)
DeprecatedDeprecated, for in-memory operations use the MEMORY_BACKEND (see User Guide section 5.14)



C++

OperationAL3AL4Remarks
creating a data-entry
(MDSplus backend)
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.create();
OR
IdsNs::IDS ids(pulse,run,0,0);
ids.createEnv(UserName, DatabaseName, IMASVersion);
IdsNs::IDS imas_entry(pulse,run,0,0)
imas_entry.createEnv(UserName, DatabaseName, IMASVersion);


opening a data-entry
(MDSplus backend)
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.open();
OR
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.openEnv(UserName, DatabaseName, IMASMajorVersion);
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.openEnv(‘usr’, ‘test’, ‘3’); 


writing the complete IDS
imas_entry._pf_active.put(); for occurrence 0
imas_entry._pf_active.put(n); for occurrence n
imas_entry._pf_active.put(); for occurrence 0
imas_entry._pf_active.put(n); for occurrence n 

writing a single slice of an IDS
imas_entry._actuator.putSlice();for occurrence 0
imas_entry._actuator.putSlice(n); for occurrence n
imas_entry._actuator.putSlice();for occurrence 0
imas_entry._actuator.putSlice(n); for occurrence n

writing only non-timed data of an IDS
imas_entry._pf_active.putNonTimed(); for
occurrence 0
imas_entry._pf_active.putNonTimed(n); for
occurrence n
DeprecatedDeprecated, replaced by put() on IDS containing only first slice data
reading the complete IDS
imas_entry._pf_active.get(); for occurrence 0
imas_entry._pf_active.get(n); for occurrence n
ids._pf_active.get(); for occurrence 0
ids._pf_active.get(n); for occurrence n

reading a single slice of an IDS
imas_entry._pf_active.getSlice(time_requested,
interpolation_method); for occurrence 0
imas_entry._pf_active.getSlice(n, time_requested,
interpolation_method); for occurrence n
imas_entry._pf_active.getSlice(time_requested, interpolation_method); for occurrence 0
imas_entry._pf_active.getSlice(n, time_requested, interpolation_method); 
for occurrence n 
CLOSEST_SAMPLE PREVIOUS_SAMPLE INTERPOLATION
copying an IDS or one of its substructure



flushing all IDSes from the cache to the disk
imas_entry.flushAll()
Deprecated

discarding all IDSes
imas_entry.discardAll()
Deprecated

deallocating an IDS or one if its substructure



checking if arrays of an IDS were allocated in C (by AL)


deleting an IDS from a data-entry
imas_entry._pf_active.remove()
imas_entry._pf_active.remove()

closing a data-entry
imas_entry.close();
imas_entry.close(); 




Python

OperationAL3AL4Remarks
creating a data-entry
(MDSplus backend)
A sequence of calls imas.ids(pulse, run) and
create() on newly created object
e.g.
import imas
ids_object = imas.ids(12,1)
ids_object.create()
sequence of calls imas.ids(pulse, run) and
e.g.
import imas
ids_object = imas.ids(12,1)
ids_object.create_env(‘usr’, ‘test’, ‘3’)
sequence of calls imas.ids(pulse, run) and
e.g.
import imas 
ids_object = imas.ids(12,1)
ids_object.create_env(‘usr’, ‘test’, ‘3’) idx = imas_create_env( treename, pulse, run, 0, 0, UserName, DatabaseName,IMASMajorVersion )


opening a data-entry
(MDSplus backend)
A sequence of calls imas.ids(pulse, run) and open()
on newly created object
e.g.
import imas
imas_entry = imas.ids(12,1,0,0)
imas_entry.open()
sequence of calls imas.ids(pulse, run) and
open_env(UserName, DatabaseName,
IMASMajorVersion) on newly created object
e.g.
import imas
imas_entry = imas.ids(12,1)

A sequence of calls imas.ids(pulse, run) and open_env(UserName, DatabaseName, IMASMajorVersion) on newly created object
e.g.
import imas
imas_entry = imas.ids(12,1)
imas_entry.open_env(‘usr’, ‘test’, ‘3’) 


writing the complete IDS


imas_entry.pf_active.put() for occurrence 0, or
imas_entry.pf_active.put(occurrence) otherwise

imas_entry.pf_active.put() for occurrence 0, or imas_entry.pf_active.put(occurrence) otherwise 

writing a single slice of an IDS

Method of the ids: def putSlice(self,
occurrence=0):
e.g.
imas_entry.pf_active.putSlice() for occurrence 0,
or imas_entry.pf_active.putSlice(occurrence)
otherwise

Method of the ids: def putSlice(self, occurrence=0):
e.g.
imas_entry.pf_active.putSlice() for occurrence 0, or imas_entry.pf_active.putSlice(occurrence) otherwise 

writing only non-timed data of an IDS

Method of the ids: def putNonTimed(self,
occurrence=0):
e.g.
imas_entry.pf_active.putNonTimed() for
occurrence 0, or
imas_entry.pf_active.putNonTimed(occurrence)
otherwise
DeprecatedDeprecated, replaced by put() on IDS containing only first slice data
reading the complete IDS

def get(self, occurrence=0):
e.g.
imas_entry.pf_active.get() for occurrence 0, or
imas_entry.pf_active.get(occurrence) otherwise

def get(self, occurrence=0):
e.g.
imas_entry.pf_active.get() for occurrence 0, or imas_entry.pf_active.get(occurrence) otherwise 

reading a single slice of an IDS

Method of the ids: def getSlice(self,
time_requested, interpolation_method,
occurrence=0):
e.g.
imas_entry.pf_active.getSlice(0.1, 2) for
occurrence 0, or
imas_entry.pf_active.getSlice(0.1, 2,
occurrence) otherwise

Method of the ids:  def getSlice(self, time_requested, interpolation_method, occurrence=0):
e.g.
imas_entry.pf_active.getSlice(0.1, 2) for occurrence 0, or imas_entry.pf_active.getSlice(0.1, 2, occurrence) otherwise 
imasdef.py:
UNDEFINED_INTERP
CLOSEST_ INTERP
PREVIOUS_INTERP  
LINEAR_INTERP  
copying an IDS or one of its substructure



flush an IDS from the cache to the disk
Method of the ids: def flushCache(self,
IDSOccurrence=0):
e.g.
25
imas_entry.pf_active.flushCache()

Deprecated

flushing all IDSes from the cache to the disk
def flushMemCache(self):
e.g.
imas_entry.flushMemCache()

Deprecated

disables temporarily the memory cache
Method of the ids: def discard(self,
IDSOccurrence=0):
e.g.
26
imas_entry.pf_active.discard()

Deprecated

discarding all IDSes
def discardMemCache(self):
eg.
imas_entry.discardMemCache()

Deprecated

deleting an IDS from a data-entry

Method of the ids: def deleteData(self,
occurrence=0):
e.g.
imas_entry.pf_active.deleteData() for
occurrence 0, or
imas_entry.pf_active.deletaData(occurrence)
otherwise

Method of the ids: def deleteData(self, occurrence=0):
e.g.
imas_entry.pf_active.deleteData() for occurrence 0, or imas_entry.pf_active.deletaData(occurrence) otherwise 

closing a data-entry
imas_entry.close();
imas_entry.close();



Matlab

OperationAL3AL4Remarks
creating a data-entry
(MDSplus backend)
idx = imas_create(treename, pulse, run)
idx = imas_create_env( treename, pulse, run, 0, 0,
UserName, DatabaseName,IMASMajorVersion )
idx = imas_create_env( treename, pulse, run, 0, 0, UserName, DatabaseName,IMASMajorVersion )


opening a data-entry
(MDSplus backend)
idx = imas_create_env( treename, pulse, run, 0, 0,
idx = imas_create(treename, pulse, run)UserName, DatabaseName,IMASMajorVersion )

  idx = imas_open_env( treename, pulse, run, UserName, DatabaseName, IMASMajorversion ) 


writing the complete IDS
 ids_put(idx, IDSOccurrence, IDSVariable)
ids_put(idx, IDSOccurrence, IDSVariable)

writing a single slice of an IDS

ids_put_slice(idx, IDOccurrence,
IDSVariable)
  ids_put_slice(idx, IDOccurrence, IDSVariable) 

writing only non-timed data of an IDS
ids_put_non_timed(idx, IDSOccurence,
IDSVariable)
Deprecated
reading the complete IDS

IDSVariable = ids_get(expIdx,
IDSOccurrence)
  IDSVariable = ids_get(expIdx, IDSOccurrence) 

reading a single slice of an IDS

IDSVariable = ids_get_slice(idx,
IDSOccurrence, time_requested,
interpolation_method)
  IDSVariable = ids_get_slice(idx, IDSOccurrence, time_requested, interpolation_method) 

copying an IDS or one of its substructure



deleting an IDS from a data-entry
 Not implemented yet
  Not implemented yet 

closing a data-entry
 [ cr ] = imas_close(idx)
  [ cr ] = imas_close(idx) 



Java

OperationAL3AL4Remarks
creating a data-entry
(MDSplus backend)
imas_entry(pulse,run,0,0);
imas_entry.create(treename, pulse, run, 0, 0);
OR
imas_entry(pulse,run,0,0);
imas_entry.createEnv (treename, pulse, run, 0, 0,UserName, DatabaseName,IMASMajorVersion)
imas.createEnv (treename, pulse, run, 0, 0, UserName, DatabaseName, IMASMajorVersion),


opening a data-entry
(MDSplus backend)
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.open();
OR
IdsNs::IDS imas_entry(pulse,run,0,0);
imas_entry.openEnv(UserName,DatabaseName,IMASMajorVersion);
imas.openEnv(treename, pulse, run, UserName, DatabaseName, IMASVersion)



writing the complete IDS
Method of the ids: put (idx, idsFullName, idsObject),
Method of the ids: put (idx, idsFullName, idsObject); 

writing a single slice of an IDS
Method of the ids: putSlice (idx, idsFullName,
idsObject)
Method of the ids: putSlice (idx, idsFullName, idsObject),

writing only non-timed data of an IDS
Method of the ids: putNonTimed (idx,
idsFullName, idsObject),
DeprecatedDeprecated, replaced by put() on IDS containing only first slice data
reading the complete IDS
Method of the ids: get (idx, idsFullName, idsObject),
Method of the ids: get (idx, idsFullName, idsObject),

reading a single slice of an IDS
Method of the ids: getSlice (idx, idsFullName,
idsObject, double time_requested, int
interpolation_method),
Method of the ids: getSlice (idx, IDSOccurrence, idsObject, double time_requested, int interpolation_method)

copying an IDS or one of its substructure



flushing all IDSes from the cache to the disk
imas_entry.flush (idx, idsFullName);

Deprecated

flushing all IDSes from the cache to the diskimas_entry.flushAll(idx);

Deprecated

disables temporarily the memory cache
imas_entry.discard (idx, idsFullName);
Deprecated

discarding all IDSesimas_entry.discardAll(idx);
Deprecated

deleting an IDS from a data-entry
Method of the ids: delete (idx, idsFullName, idsObject)
Method of the ids: delete (idx, idsFullName, idsObject),

closing a data-entry
imas.close(idx) ;
 imas.close(idx) ; 



Lowlevel and backend 
A part of the new low level API (4.x) is exposed through the high-level interfaces and can be used to select the backend being used to store the data (or to read the data from), amongst the currently available MDSPLUS, MEMORY, UDA and ASCII. As the lowlevel API is flexible and generic, three functions need to be called in that context. Refer to AL User's Guide for description of each functions in the respective HLI language.
Lowlevel (C)