1. High Level Interfaces and their API (Application Programming Interface)
The IMAS Data Access Layer exposes a couple of operations (so-called API) for writing or reading IDSs data.
These data access operations are available from users code in the currently supported programming languages, the so-called High Level Interfaces (HLI):
- Fortran
- C++
- Java
- Python
- Matlab
The HLI API can be divided into 2 sets of operations:
- operations applying on a Data Entry
- operations applying on an IDS
The HLI API covers all available Access Layer features with the following exposed methods:
- —Operations on a Data Entry:
- —CREATE (creation of a new Data Entry)
- —OPEN (opening an existing Data Entry)
- —CLOSE (closing a Data Entry)
- —Operations on an IDS:
- —PUT (writing data from an IDS to a Data Entry)
- —GET (reading data of an IDS from an existing Data Entry)
- —PUT_SLICE (writing a IDS time slice to a Data Entry)
- —GET_SLICE (reading a time slice of an IDS from an existing Data Entry)
- —DELETE (deleting an IDS from an existing Data Entry)
In this tutorial, we will describe each method of the HLI API (section 1.1. HLI API). We will use the Python HLI. Documentation of all others HLIs is available in the User guide available from this page: https://confluence.iter.org/display/IMP/Integrated+Modelling+Home+Page
1.1. HLI API (Ludovic)
1.1.1. create/close
The creation of a new Data Entry using the MDS+ backend:
- consists in creating a new (MDS+) pulse file on the disk
- requires to have an existing 'database' (corresponding to a directory when using the MDS+ backend) which will host the new pulse file
So, let's first create a new database named 'data_access_tutorial' which will belong to the current user.
From a new shell, execute the following command:
module load IMAS imasdb data_access_tutorial
Let's check that the database has been successfully created:
$ ls -alh ~/public/imasdb/data_access_tutorial/ total 2.5K drwxrwsr-x 3 fleuryl fleuryl 4.0K Aug 31 10:09 . drwxrwsr-x 14 fleuryl fleuryl 4.0K Aug 31 10:09 .. drwxrwsr-x 13 fleuryl fleuryl 4.0K Sep 13 11:16 3
Some sub-directories have been created, they are used by MDS+ to organize the pulse files.
Now, the following code will create a new MDS+ pulse file for shot=15000, run=1 in the 'data_access_tutorial' database of the current user:
import imas import getpass from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'pcss_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #creates the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.create() #here, we can perform some read/write operations using the get/put() operations #... we will #closes the data_entry data_entry.close()
Execution of the code above will create the pulse file at location ~/public/imasdb/data_access_tutorial/3/0:
$ ls -alh ~/public/imasdb/data_access_tutorial/3/0 total 78M drwxrwsr-x 2 fleuryl fleuryl 4.0K Aug 31 10:09 . drwxrwsr-x 12 fleuryl fleuryl 4.0K Aug 31 10:09 .. -rw-rw-r-- 1 fleuryl fleuryl 42M Aug 31 10:09 ids_150000001.characteristics -rw-rw-r-- 1 fleuryl fleuryl 37 Aug 31 10:09 ids_150000001.datafile -rw-rw-r-- 1 fleuryl fleuryl 36M Aug 31 10:09 ids_150000001.tree
In the example above, the pulse file is created then closed. However no data have been yet saved to the pulse file.
1.1.2. open/close
The following code opens the existing MDS+ pulse file created previously for shot=15000, run=1, from the 'data_access_tutorial' database of the current user:
import imas import getpass from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.open() #here, we can perform some read/write operations using the get/put() operations #... #closes the data_entry data_entry.close()
The existing pulse file is opened then closed. However no data have been yet saved to the pulse file.
1.1.3. get
IMAS data are contained in IDSs which are containers described by the IMAS Data Dictionary (DD). An IDS represents either a tokamak subsystem (like 'camera_ir'), or a concept like the 'equilibrium' IDS representing a plasma equilibrium.
An IDS can contain 0D (scalar) data with integer, float or string type, and arrays of integers or floats with dimensions from 1 to 6.
As an example, let's consider the 'magnetics' IDS. It's description is given by the Data Dictionary. Here is a snapshot:
Like others IDSs, the 'magnetics' IDS contains data structures (like 'ids_properties') and array of structures (like 'flux_loop', 'bpol_probe', 'b_field_pol_probe', ...) which can contain (float, integer) data arrays with 1 to 6 dimensions, or scalars (0D).
Let's focus on the 'flux_loop' array of structures (see snapshot below). It contains an array of 'flux' structures.
Each flux structure contains a FLT_1D data array named 'data', a INT_1D array named ' validity_timed', a INT_0D scalar named 'validity', a FLT_1D data array named 'time'.
FLT_1D designates a 1 dimension array containing floats, INT_1D designates a 1 dimension array containing integers and INT_0D designates an integer scalar.
Each IDS exposes the get() operation which reads all IDS data from an opened data_entry.
When calling the get() operation on a IDS, all scalars and data arrays contained in the IDS are read. All these data are put in memory.
The code below reads an existing 'magnetics' IDS from a WEST pulse file:
import imas
import getpass
import numpy as np
from imas import imasdef
#opens the Data Entry object 'data_entry' associated to the pulse file with shot=54178, run=0, belonging to database 'west' of user 'g2lfleur', using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west, 54178, 0, user_name='g2lfleur')
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
#reads the 'magnetics' IDS from the data_entry object previously opened
magnetics_ids = data_entry.get('magnetics', 0) #The second argument 0 is the so-called IDS occurrence.
#close the pulse file associated to the 'data_entry' object
data_entry.close()
#prints some IDS attributes
print('Number of flux loops = ', len(magnetics_ids.flux_loop))
print('Data of first flux loop = ', magnetics_ids.flux_loop[0].flux.data)
print('Homogeneous time basis = ', magnetics_ids.time)
Running the code above gives the following output:
Number of flux loops = 17 Data of first flux loop = [ 0.00065229 0.00163073 0.00489218 ... -0.01761185 -0.01663342 -0.01500269] Homogeneous time basis = [ 1.83570397 1.86847198 1.90123999 ... 90.13289642 90.16566467 90.19843292]
1.1.4. put
Let's create and initialize a new 'magnetics' IDS and let's add it to the pulse file previously created in section 1.1.1.
Then, in order to write all data (scalars and data arrays) contained in the newly created IDS to the pulse file created previously, we will use the put() operation which writes all static (non time dependent) AND dynamic data present in the IDS.
The code below:
- opens the Data Entry created in section 1.1.1.
- create a 'magnetics' IDS object
- intializes the 'magnetics' IDS object with some values (some are mandatory)
- adds the IDS data to the Data Entry calling the put() operation
import imas import getpass import numpy as np from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.open() #creates the 'magnetics' IDS and initializes it magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS magnetics_ids.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory) magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute magnetics_ids.time=np.array([0]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime #writes the 'magnetics' IDS data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence. #closes the pulse file associated to the 'data_entry' object data_entry.close()
Let's extend the above example by adding the WEST data of the 10 first flux loops to the newly created 'magnetics' IDS.
import imas
import getpass
import numpy as np
from imas import imasdef
#creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser())
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
#creates the 'magnetics' IDS and initializes it
magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS
magnetics_ids.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)
magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
#adding the WEST data of the 10 first flux loops
nb_flux_loops = 10
west_data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west', 54178, 0, 'g2lfleur')
west_magnetics_ids = west_data_entry.get('magnetics', 0) #reading occurrence 0
magnetics_ids.flux_loop.resize(nb_flux_loops)
for i in range(nb_flux_loops):
magnetics_ids.flux_loop[i].flux.data = west_magnetics_ids.flux_loop[i].flux.data #copies data
if west_magnetics_ids.ids_properties.homogeneous_time==0:
magnetics_ids.flux_loop[i].flux.time = west_magnetics_ids.flux_loop[i].flux.time #copies the time basis in case WEST IDS arrays don't accept a common time basis
if west_magnetics_ids.ids_properties.homogeneous_time==1:
magnetics_ids.time = west_magnetics_ids.time #copies the 'root' time basis in case WEST IDS arrays accept a common time basis
west_data_entry.close() #closing the WEST pulse file
#writes the 'magnetics' IDS
data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence.
#closes the pulse file associated to the 'data_entry' object
data_entry.close()
1.1.5. put_slice
An IDS containg dynamic data structures can be built progressively using data time slices.
A dynamic data structure:
- is either an array with type INT_nD or FLT_nD where n=1 to 6, which accepts a time coordinate
- or a so-called dynamic array of structures AOS[i] where i runs along a time index
In the example below, we illustrate the use of put_slice() on a 'camera_visible' IDS which contains the dynamic array of structures 'frame'.
Let's first have a look to the Data Dictionary of the 'camera_visible' IDS description:
As indicated in the DD description, 'frame(itime)' is a dynamic data structure representing a set of frames.
Each element of 'frame' is a structure containing:
- 'image_raw', a INT_2D array
- 'radiance', a FLT_2D array
- 'time', a time basis (used only in case the IDS has no common time basis)
Adding a time slice using put_slice() on the 'camera_visible' will have for effect to add a new element to the dynamic array 'frame', that is, a new frame for a given time is appended to the array of structures 'frame'.
The example below shows how to add few time slices to the 'camera_visible' IDS, for channel 0 and detector 0:
- The number of time slices to be appended is given by the variable 'nb_slices'.
- Each time slice is represented by the data structure: camera_visible_ids.channel[0].detector[0].frame[0]
- Only the data of the INT_2D 'image_raw" and the data of global time basis 'time' (since the 'camera_visible' has homogeneous_time = 1) are populated in the time slice to be appended
- The first time slice needs to be appended using a call to put(), not put_slice(). Only subsequent time slices are appended using put_slice().
Time slices must be appended in strictly increasing time order
import imas import getpass import numpy as np from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the data_entry object data_entry.open() #creates the 'magnetics' IDS and initializes it camera_visible_ids = imas.camera_visible() camera_visible_ids.ids_properties.homogeneous_time = 1 camera_visible_ids.channel.resize(1) #using only 1 channel (channel 0) for this example camera_visible_ids.channel[0].detector.resize(1) #using only 1 detector for channel 0 camera_visible_ids.channel[0].detector[0].frame.resize(1) #the array of structure 'frame' contains only 1 element, it is the slice to be appended to the IDS X = 3 #number of horizontal pixels in the frame Y = 5 #number of vertical pixels in the frame camera_visible_ids.channel[0].detector[0].frame[0].image_raw.resize(X,Y) #setting the size of the image of the frame camera_visible_ids.time.resize(1) #the time vector contains only 1 element, it is the time of the slice nb_slices=10 #number of time slices to be added for i in range(nb_slices): camera_visible_ids.time[0] = float(i) #time of the slice for j in range(X): for k in range(Y): camera_visible_ids.channel[0].detector[0].frame[0].image_raw[j,k] = float(j + k + i) #image_raw is a 2D array containing the data (pixels) of the frame if i==0: data_entry.put(camera_visible_ids) #the first frame has to be added using put() in order to store static data as well else: data_entry.put_slice(camera_visible_ids) #appending the the slice to the IDS #closes the pulse file associated to the 'data_entry' object data_entry.close()
Let's check the time slices we have just appended to the 'camera_visible' IDS using the code below:
import imas
import getpass
from imas import imasdef
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
#reads the 'magnetics' IDS data using get()
camera_visible_ids= data_entry.get('camera_visible',0)
#prints some IDS attributes
print('homogeneous_time = ', camera_visible_ids.ids_properties.homogeneous_time)
for i in range(3):
print("Frame : ", i)
print(camera_visible_ids.channel[0].detector[0].frame[i].image_raw) #prints the content of this 2D array
print("-----")
#closes the pulse file associated to the 'data_entry' object
data_entry.close()
Running the code above gives the following ouptut:
homogeneous_time = 1 Time slice : 0.0 Image raw: [[0 1 2 3 4] [1 2 3 4 5] [2 3 4 5 6]] ----- Time slice : 1.0 Image raw: [[1 2 3 4 5] [2 3 4 5 6] [3 4 5 6 7]] ----- Time slice : 2.0 Image raw: [[2 3 4 5 6] [3 4 5 6 7] [4 5 6 7 8]] -----
1.1.6. get_slice
The get_slice() operation has the following signature:
get_slice(<idsname>, time_requested, interpolation_method, occurrence = 0)
Let's consider an IDS containing dynamic data as the 'camera_visible' IDS described above.
Calling get_slice('camera_visible', t, interpolation_method, 0) will take a slice (at given time 't' and for a given interpolation method) of each dynamic data structure contained in the IDS, static data structures are ignored.
Therefore, get_slice() returns a time slice over all IDS dynamic data structures.
The following code takes a slice of the IDS dynamic data at time=1s:
import imas
import getpass
import numpy as np
from imas import imasdef
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser())
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
time_requested=1.
slice = data_entry.get_slice('camera_visible', time_requested, imasdef.CLOSEST_INTERP)
print("Slice time : ", time_requested)
print("Image raw:")
print(slice.channel[0].detector[0].frame[0].image_raw)
print("-----")
data_entry.close()
Running the code above gives the following output:
Slice time : 1.0 Image raw: [[1 2 3 4 5] [2 3 4 5 6] [3 4 5 6 7]] -----