High Level Interfaces & their Application Programming Interface - PDF presentation

High Level Interfaces and their API (Application Programming Interface)

...

The HLI API covers all available Access Layer features with the following exposed methods:

...

In this tutorial, we will describe each method of the HLI API (section 1.1. HLI API). We will use the Python HLI.  

API functions name and their signature may differ from one HLI to another. Documentation of all others HLIs is available in the User guide available from this page: https://confluence.iter.org/display/IMP/Integrated+Modelling+Home+Page

...

The creation of a new Data Entry using the MDS+ or HDF5 backend:

• consists in creating a new (MDS+/HDF5) pulse file on the disk
• requires to have at least an existing 'database' (corresponding to a directory when using the MDS+ backend) which will host the new pulse filefor hosting pulse file(s)

So, let's first create a new database named 'data_access_tutorial' which will belong to the current user.

...

Let's check that the database (it's simply a directory with some subdirectories for MDS+) has been successfully created:

<g2lfleur@s52 ~>ls -alh ~/public/imasdb/data_access_tutorial/
total 6.0K
drwxr-xr-x  3 g2lfleur g2itmdev 2.0K Sep 16 13:29 .
drwxr-xr-x  5 g2lfleur g2itmdev 2.0K Sep 16 13:29 ..
drwxr-xr-x 12 g2lfleur g2itmdev 2.0K Sep 16 13:29 3

...

import imas
import getpass
from imas import imasdef

#creates#creating the Data Entry object 'data_entry', a kind of handler ofwhich handles the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser())

#creates#creating the pulse file associatedhandled toby the Data Entry object 'data_entry' previously created
data_entry.create()

#here,#we wecould cannow perform some write operations using the put() operation, it
#...This will be dealt with later
#... 

#closes#closing the data_entryData Entry
data_entry.close()

Execution of the code above will create the pulse file at location ~/public/imasdb/data_access_tutorial/3/0:

<g2lfleur@s52 ~>ls -alh ~/public/imasdb/data_access_tutorial/3/0/
total 78M
drwxr-xr-x  2 g2lfleur g2itmdev 2.0K Sep 16 15:28 .
drwxr-xr-x 12 g2lfleur g2itmdev 2.0K Sep 16 13:29 ..
-rw-r--r--  1 g2lfleur g2itmdev  42M Sep 16 15:28 ids_150000001.characteristics
-rw-r--r--  1 g2lfleur g2itmdev    0 Sep 16 15:28 ids_150000001.datafile
-rw-r--r--  1 g2lfleur g2itmdev  36M Sep 16 15:28 ids_150000001.tree

In the above example above, the pulse file is created then closed. However no data have been yet saved to the pulse file (the file ids_150000001.datafilefile is empty).

open/close

The following code opens the existing MDS+ pulse file created previously for shot=15000, run=1, from the 'data_access_tutorial' database of the current user:

import imas
import getpass
from imas import imasdef

#creates the Data Entry object 'data_entry' associated  to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser())

#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()

#here, we can perform some read/write operations using get/put() or get_slice()/put_slice() operations
#...

#closes the data_entry
data_entry.close()

 The existing pulse file is opened then closed. However no data have been yet saved to the pulse file.

get

IMAS data are contained in IDSs which are containers described by the IMAS Data Dictionary (DD). An IDS represents either a tokamak subsystem (like 'camera_ir'), or a concept like the 'equilibrium' IDS representing a plasma equilibrium.

An IDS can contain 0D (scalar) data with integer, float or string type, and arrays of integers or floats with dimensions from 1 to 6. 

As an example, let's consider the 'magnetics' IDS. It's description is given by the Data Dictionary. Here is a snapshot:

Image Removed

Like others IDSs, the 'magnetics' IDS contains data structures (like 'ids_properties') and array of structures (like 'flux_loop', 'bpol_probe', 'b_field_pol_probe', ...) which can contain (float, integer) data arrays with 1 to 6 dimensions, or scalars (0D). 

Let's focus on the 'flux_loop' array of structures (see snapshot below). It contains an array of 'flux' structures.

Each flux structure contains a FLT_1D data array named 'data', a INT_1D array named ' validity_timed', a INT_0D scalar named 'validity', a FLT_1D data array named 'time'.

FLT_1D designates a 1 dimension array containing floats, INT_1D designates a 1 dimension array containing integers and INT_0D designates an integer scalar.  

Image Removed

Each IDS exposes the get() operation which reads all IDS data from an opened data_entry.

When calling the get() operation on a IDS, all scalars and data arrays contained in the IDS are read. All these data are put in memory. 

The code below reads an existing 'magnetics' IDS from a WEST pulse file:

open/close

The following code opens the existing MDS+ pulse file created previously for shot=15000, run=1, from the 'data_access_tutorial' database of the current user:

import imas
import getpass
from imas import imasdef

#creating the Data Entry which handles pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1)

#opening the pulse file handled by the Data Entry object previously created
data_entry.open()

#we could now perform some read/write operations using get/put() or get_slice()/put_slice() operations
#...This will be dealt with later

#closing the Data Entry
data_entry.close()

 The existing pulse file is opened then closed. However no data have been yet saved to the pulse file.

get

IMAS data are contained in IDSs which are containers described by the IMAS Data Dictionary (DD). An IDS represents either a tokamak subsystem (like 'camera_ir'), or a concept like the 'equilibrium' IDS representing a plasma equilibrium.

An IDS can contain 0D (scalar) data with integer, float or string type, and arrays of integers or floats with dimensions from 1 to 6. 

As an example, let's consider the 'magnetics' IDS. It's description is given by the Data Dictionary. Here is a snapshot:

Image Added

Like others IDSs, the 'magnetics' IDS contains data structures (like 'ids_properties') and array of structures (like 'flux_loop', 'bpol_probe', 'b_field_pol_probe', ...) which can contain (float, integer) data arrays with 1 to 6 dimensions, or scalars (0D). 

Let's focus on the 'flux_loop' array of structures (see snapshot below). It contains an array of 'flux' structures.

Each flux structure contains a FLT_1D data array named 'data', a INT_1D array named ' validity_timed', a INT_0D scalar named 'validity', a FLT_1D data array named 'time'.

FLT_1D designates a 1 dimension array containing floats, INT_1D designates a 1 dimension array containing integers and INT_0D designates an integer scalar.  

Image Added

Each IDS exposes the get() operation which reads all IDS data from an opened data_entry.

When calling the get() operation on a IDS, all scalars and data arrays contained in the IDS are read. All these data are put in memory. 

The code below reads an existing 'magnetics' IDS from a WEST pulse file:

import imas
import getpass
import numpy as np
from imas import imasdef

#opening the Data Entry which handles the pulse file with shot=

import imas
import getpass
import numpy as np
from imas import imasdef

#opens the Data Entry object 'data_entry' associated  to the pulse file with shot=54178, run=0, belonging to database 'west' of user 'g2lfleur', using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west, 54178, 0, user_name='g2lfleur')

#opens#opening the pulse file associatedhandled toby the Data Entry object 'data_entry' previously created
data_entry.open()

#reads#reading the 'magnetics' IDS from the data_entry object previously opened
magnetics_ids = data_entry.get('magnetics', 0) #The second argument 0 is the so-called IDS occurrence.

#close#closing the pulse file associated to the 'data_entry' objectData Entry
data_entry.close()

#prints#printing some IDS attributes
print('Number of flux loops = ', len(magnetics_ids.flux_loop))
print('Data of first flux loop = ', magnetics_ids.flux_loop[0].flux.data)
print('Homogeneous time basis = ', magnetics_ids.time)

...

Number of flux loops =  17
Data of first flux loop =  [ 0.00065229  0.00163073  0.00489218 ... -0.01761185 -0.01663342
 -0.01500269]
Homogeneous time basis =  [ 1.83570397  1.86847198  1.90123999 ... 90.13289642 90.16566467
 90.19843292]

put

put

Let's create and initialize a new 'Let's create and initialize a new 'magnetics' IDS and let's add it to the pulse file previously created in section 1.1.1. 

...

• opens the Data Entry created in section 1.1.1.
• create creates a 'magnetics' IDS object
• intializes the 'magnetics' IDS object with some values (some are mandatory)
• adds the IDS data to the Data Entry calling the put() operation

...

import imas
import getpass
import numpy as np
from imas import imasdef

#creates#creating the Data Entry object 'data_entry' associated  to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())

#opens#opening the pulse file associatedhandled toby the Data Entry object 'data_entry' previously created
data_entry.open() 

#creates#creating the 'magnetics' IDS and initializesinitializing it
magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS
magnetics_ids.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)
magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
magnetics_ids.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime

#writes#writing the 'magnetics' IDS
data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence.

#closes#closing the pulse file associated to the 'data_entry' objectData Entry
data_entry.close() 	 

Let's extend the above example by adding the WEST data of the 10 first flux loops to the newly created 'magnetics' IDS.

import imas
import getpass
import numpy as np
from imas import imasdef

#creates#creating the Data Entry object 'data_entry' associated  to the pulse filewhich handles the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())

#opens#opening the pulse file associatedhandled toby the Data Entry object 'data_entry' previously created
data_entry.open() 

#creates#creating the 'magnetics' IDS and initializesinitializing it
magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS
magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute

#adding the WEST data of the 10 first flux loops
nb_flux_loops = 10
west_data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west', 54178, 0, 'g2lfleur')
west_magnetics_ids = west_data_entry.get('magnetics', 0) #reading occurrence 0
magnetics_ids.ids_properties.homogeneous_time=west_magnetics_ids.ids_properties.homogeneous_time #setting the homogeneous time (mandatory)
magnetics_ids.flux_loop.resize(nb_flux_loops)
for i in range(nb_flux_loops):
  magnetics_ids.flux_loop[i].flux.data = west_magnetics_ids.flux_loop[i].flux.data #copies data
  if west_magnetics_ids.ids_properties.homogeneous_time==0:
    magnetics_ids.flux_loop[i].flux.time = west_magnetics_ids.flux_loop[i].flux.time #copies the time basis in case WEST IDS arrays don't accept a common time basis

if west_magnetics_ids.ids_properties.homogeneous_time==1:
   magnetics_ids.time = west_magnetics_ids.time #copies the 'root' time basis in case WEST IDS arrays accept a common time basis
   
west_data_entry.close() #closing the WEST pulse file

#writes#writing the 'magnetics' IDS
data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence.

#closes#closing the pulse file associated to the 'data_entry' objectData Entry
data_entry.close() 	 

put_slice

...

• The number of time slices to be appended is given by the variable 'nb_slices'.
• Each time slice is represented by the data structure: camera_visible_ids.channel[0].detector[0].frame[0]
• Only In this example, only the data of the INT_2D 'image_raw" and the data of global time basis 'time' (since the 'camera_visible' has homogeneous_time = 1) are populated in the time slice to be appended

...

import imas
import getpass
import numpy as np
from imas import imasdef

#creates#creating the Data Entry object 'data_entry' associated  to which handles the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser()))

#opens#opening the pulse file associatedhandled toby the data_entryData Entry object previously created
data_entry.open()

#creates#creating the 'magneticscamera_visible' IDS and initializesinitializing it
camera_visible_ids = imas.camera_visible()
camera_visible_ids.ids_properties.homogeneous_time = 1
camera_visible_ids.channel.resize(1) #using only 1 channel (channel 0) for this example
camera_visible_ids.channel[0].detector.resize(1) #using only 1 detector for channel 0
camera_visible_ids.channel[0].detector[0].frame.resize(1) #the array of structure 'frame' contains only 1 element, it is the slice to be appended to the IDS 

X = 3 #number of horizontal pixels in the frameof 2D 'image_raw' field
Y = 5 #number of vertical pixels of in the frame2D 'image_raw' field

camera_visible_ids.channel[0].detector[0].frame[0].image_raw.resize(X,Y) #setting the size of the image of the frame
camera_visible_ids.time.resize(1) #the time vector contains only 1 element, it's is the time of the slice

nb_slices=103 #number of time slices to be added

for i in range(nb_slices):
  camera_visible_ids.time[0] = float(i) #time of the slice
  for j in range(X):
    for k in range(Y):
       camera_visible_ids.channel[0].detector[0].frame[0].image_raw[j,k] = float(j + k +  i) #image_raw is a 2D array
  containing the data (pixels) of the frame
  if i==if i==0:
    data_entry.put(camera_visible_ids) #the first frameslice has to be added using put() in order to store static data as well
  else:
    data_entry.put_slice(camera_visible_ids)  #appending the thecurrent slice to the IDS

#closes the pulse file associated to the 'data_entry' object 'frame' array of structures

#closing the Data Entry
data_entry.close() 	 

Let's check the time slices we have just appended to the 'camera_visible' IDS using the code below:

import imas
import getpass
from imas import imasdef

data_entry#creating = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())

#opensthe Data Entry object which handles the pulse file associated to the Data Entry objectwith shot=15000, run=1, belonging to database 'data_entry' previously created
data_entry.open()

#reads the 'magnetics' IDS data using get()
camera_visible_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1)

#opening the pulse file handled by the Data Entry object previously created
data_entry.open()

#reading the 'camera_visible' IDS data using get()
camera_visible_ids= data_entry.get('camera_visible',0)

#prints#printing some IDS attributes
print('homogeneous_time = ', camera_visible_ids.ids_properties.homogeneous_time)

for i in range(3):
   print("Frame : ", i)
   print(camera_visible_ids.channel[0].detector[0].frame[i].image_raw) #prints the content of this 2D array
   print("-----")

#closes#closing the pulse file associated to the 'data_entry' objectData Entry
data_entry.close() 	 

Running the code above gives the following ouptut:

...

Calling get_slice('camera_visible', t, interpolation_method, 0) will take a slice (at given time 't' and for a given interpolation method) of each dynamic data structure contained in the IDS, static data structures are ignored.

Therefore, get_slice() returns a time slice over all IDS dynamic data structures.

the IDS, static data structures are ignored.

Therefore, get_slice() returns a time slice over all IDS dynamic data structures.

The following code takes a slice of the IDS dynamic data at time=1s using the closest time slice interpolation:

import imas
import getpass
import numpy as np
from imas import imasdef

#creating the Data Entry object which handles the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1)

#opening the pulse file handled by the Data Entry object previously created
data_entry.open()

#getting a slice at time=1s using the closest time slice interpolation
time_requested=1.
slice = data_entry.get_slice('camera_visible', time_requested, imasdef.CLOSEST_INTERP)

print("Slice time : ", time_requested)
print("Image raw:")
print(slice.channel[0].detector[0].frame[0].image_raw)
print("-----")

#closing the Data Entry
data_entry.close()

Running the code above gives the following output:

Slice time :  1.0
Image raw:
[[1 2 3 4 5]
 [2 3 4 5 6]
 [3 4 5 6 7]]
-----

 delete_data

A specific occurrence of an IDS can be deleted from an existing Data Entry using the delete_data() operation.

The following code erases the occurrence 0 of the 'magnetics' IDS previously created in previous sectionsThe following code takes a slice of the IDS dynamic data at time=1s:

import imas
import getpass
import numpy as np
from imas import imasdef
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())
#opens
#creating the Data Entry object which handles the pulse file associated to the Data Entry objectwith shot=15000, run=1, belonging to database 'data_access_entrytutorial' of the previously created
data_entry.open()

time_requested=1.
slice = data_entry.get_slice('camera_visible', time_requested, imasdef.CLOSEST_INTERP)

print("Slice time : ", time_requested)
print("Image raw:")
print(slice.channel[0].detector[0].frame[0].image_raw)
print("-----")

data_entry.close()

Running the code above gives the following output:

Slice time :  1.0
Image raw:
[[1 2 3 4 5]
 [2 3 4 5 6]
 [3 4 5 6 7]]
-----

...

current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1)

#removing occurrence 0 of the 'magnetics' IDS previously appended to the Data Entry
data_entry.delete_data('magnetics', 0) #opens previously the Data Entry if it is closed, then delete occurrence 0 of the 'magnetics' IDS

#closing the Data Entry 
data_entry.close()