1. High Level Interfaces and their API (Application Programming Interface)
The IMAS Data Access Layer exposes a couple of operations (so-called API) for writing or reading IDSs data.
These data access operations are available from users code in the currently supported programming languages, the so-called High Level Interfaces (HLI):
- Fortran
- C++
- Java
- Python
- Matlab
The HLI API can be divided in 2 sets of operations:
- operations applying on a Data Entry
- operations applying on an IDS
The HLI API covers all available Access Layer features with the following exposed methods:
- —Operations on a data base entry:
- —CREATE (creation of a new Data Entry)
- —OPEN (opening an existing Data Entry)
- —CLOSE (closing a Data Entry)
- —Operations on IDSs:
- —PUT (writing data from an IDS to a Data Entry)
- —GET (reading data of an IDS from an existing Data Entry)
- —PUT_SLICE (writing a IDS time slice to a Data Entry)
- —GET_SLICE (reading a time slice of an IDS from an existing Data Entry)
- —DELETE (deleting an IDS from an existing Data Entry)
In this tutorial, we will describe each method of the HLI API (section 1.1. HLI API). We will use the Python HLI. Documentation of all others HLIs is available in the User guide available from this page: https://confluence.iter.org/display/IMP/Integrated+Modelling+Home+Page
1.1. HLI API (Ludovic)
1.1.1. create/close
Creating a new Data Entry using the MDS+ backend:
- consists in creating a new (MDS+) pulse file on the disk
- requires to have an existing 'database' (e.g a directory when using the MDS+ backend) which will host the new pulse file
So, let's first create a new database named 'data_access_tutorial' which will belong to the current user.
From a new shell, execute the following command:
module load IMAS imasdb data_access_tutorial
We can check that the database has been successfully created:
$ ls -alh ~/public/imasdb/data_access_tutorial/ total 2.5K drwxrwsr-x 3 fleuryl fleuryl 4.0K Aug 31 10:09 . drwxrwsr-x 14 fleuryl fleuryl 4.0K Aug 31 10:09 .. drwxrwsr-x 13 fleuryl fleuryl 4.0K Sep 13 11:16 3
Now, the following code will create a new MDS+ pulse file for shot=15000, run=1 in the 'data_access_tutorial' database of the current user:
import imas import getpass from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'pcss_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #creates the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.create() #here, we can perform some read/write operations using the get/put() operations #... #closes the data_entry data_entry.close()
Execution of the code above will create the pulse file at location ~/public/imasdb/data_access_tutorial/3/0:
$ ls -alh ~/public/imasdb/data_access_tutorial/3/0 total 78M drwxrwsr-x 2 fleuryl fleuryl 4.0K Aug 31 10:09 . drwxrwsr-x 12 fleuryl fleuryl 4.0K Aug 31 10:09 .. -rw-rw-r-- 1 fleuryl fleuryl 42M Aug 31 10:09 ids_150000001.characteristics -rw-rw-r-- 1 fleuryl fleuryl 37 Aug 31 10:09 ids_150000001.datafile -rw-rw-r-- 1 fleuryl fleuryl 36M Aug 31 10:09 ids_150000001.tree
In the example above, the pulse file is created, then closed. However no data have been yet saved to the pulse file.
1.1.2. open/close
The following code opens the existing MDS+ pulse file created previously for shot=15000, run=1, from the 'data_access_tutorial' database of the current user:
import imas import getpass from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.open() #here, we can perform some read/write operations using the get/put() operations #... #closes the data_entry data_entry.close()
In the example above, the pulse file is opened, then closed. However no data have been yet fetched from the pulse file.
1.1.3. get
IMAS data are contained in IDSs which are containers described by the IMAS Data Dictionary. An IDS represents either a tokamak subsystem (like 'camera_ir'), or a concept like the 'equilibrium' IDS representing a plasma equilibrium.
An IDS can contain 0D (scalar) data and arrays with dimensions from 1 to 6.
As an example, let's consider the 'magnetics' IDS. It's description is given by the Data Dictionary. Here is a snapshot:
Like others IDSs, the 'magnetics' IDS contains data structures (like 'ids_properties') and array of structures (like 'flux_loop', 'bpol_probe', 'b_field_pol_probe', ...) which can contain (float, integer) data arrays with 1 to 6 dimensions, or scalars (0D).
Let's focus on the 'flux_loop' array of structures (see snapshot below). It contains an array of 'flux' structures.
Each flux structure contains a FLT_1D data array named 'data', a INT_1D array named ' 'validity_timed', a INT_0D scalar named 'validity', a FLT_1D data array named 'time'.
FLT_1D designates a 1 dimension array containing floats, INT_1D designates a 1 dimension array containing integers and INT_0D designates an integer scalar.
An IDS exposes the get() operation which reads all IDS data from an opened data_entry (see above).
When calling the get() operation on a IDS, all scalars and data arrays contained in the IDS are read. All these data are put in memory.
The code below reads an existing 'magnetics' IDS from a WEST pulse file:
import imas
import getpass
import numpy as np
from imas import imasdef
#opens the Data Entry object 'data_entry' associated to the pulse file with shot=54178, run=0, belonging to database 'west' of user 'g2lfleur', using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west, 54178, 0, user_name=getpass.getuser())
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
#reads the 'magnetics' IDS from the data_entry object previously opened
magnetics_ids = data_entry.get('magnetics', 0) #The second argument 0 is the so-called IDS occurrence.
#close the pulse file associated to the 'data_entry' object
data_entry.close()
#prints some IDS attributes
print('Number of flux loops = ', len(magnetics_ids.flux_loop))
print('Data of first flux loop = ', magnetics_ids.flux_loop[0].flux.data)
print('Homogeneous time basis = ', magnetics_ids.time)
Running the code above gives the following output:
Number of flux loops = 17 Data of first flux loop = [ 0.00065229 0.00163073 0.00489218 ... -0.01761185 -0.01663342 -0.01500269] Homogeneous time basis = [ 1.83570397 1.86847198 1.90123999 ... 90.13289642 90.16566467 90.19843292]
1.1.4. put
In order to write all data (scalars and data arrays) contained in a IDS to the pulse file created previously, we will use the put() operation which writes all static (non time dependent) AND dynamic data present in the IDS.
Let's add a 'magnetics' IDS to the pulse file previously created.
The first part of the code below is opening a data_entry then a magnetics IDS is created and written to the data_entry using the put() operation:
import imas import getpass import numpy as np from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the Data Entry object 'data_entry' previously created data_entry.open() #creates the 'magnetics' IDS and initializes it magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS magnetics_ids.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory) magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute magnetics_ids.time=np.array([0]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime #writes the 'magnetics' IDS data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence. #closes the pulse file associated to the 'data_entry' object data_entry.close()
Let's extend the above example by adding the WEST data of the 10 first flux loops to the newly created 'magnetics' IDS.
import imas
import getpass
import numpy as np
from imas import imasdef
#creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial, 15000, 1, user_name=getpass.getuser())
#opens the pulse file associated to the Data Entry object 'data_entry' previously created
data_entry.open()
#creates the 'magnetics' IDS and initializes it
magnetics_ids = imas.magnetics() #creates a 'magnetics' IDS
magnetics_ids.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)
magnetics_ids.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
magnetics_ids.time=np.array([0]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime
#adding the WEST data of the 10 first flux loops
nb_flux_loops = 10
west_data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'west', 54178, 0, 'g2lfleur')
west_magnetics_ids = west_data_entry.get('magnetics', 0) #reading occurrence 0
magnetics_ids.flux_loop.resize(nb_flux_loops)
for i in range(nb_flux_loops):
magnetics_ids.flux_loop[i].flux.data = west_magnetics_ids.flux_loop[i].flux.data #copies data
west_data_entry.close() #closing the WEST pulse file
#writes the 'magnetics' IDS
data_entry.put(magnetics_ids, 0) #writes magnetics data to the data_entry associated to the pulse file. The second argument 0 is the so-called IDS occurrence.
#closes the pulse file associated to the 'data_entry' object
data_entry.close()
1.1.5. put_slice
import imas import getpass import numpy as np from imas import imasdef #creates the Data Entry object 'data_entry' associated to the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, 'data_access_tutorial', 15000, 1, user_name=getpass.getuser()) #opens the pulse file associated to the data_entry object data_entry.open() #creates the 'magnetics' IDS and initializes it camera_visible_ids = imas.camera_visible() camera_visible_ids.ids_properties.homogeneous_time = 1 camera_visible_ids.channel.resize(1) #using only 1 channel (channel 0) for this example camera_visible_ids.channel[0].detector.resize(1) #using only 1 detector for channel 0 camera_visible_ids.channel[0].detector[0].frame.resize(1) #the array of structure 'frame' contains only 1 element, it is the frame to be appended to the IDS X = 3 #number of horizontal pixels in the frame Y = 5 #number of vertical pixels in the frame camera_visible_ids.channel[0].detector[0].frame[0].image_raw.resize(X,Y) #setting the size of the image of the frame camera_visible_ids.time.resize(1) #the time vector contains only 1 element, it is the time of the slice nb_frames=10 #number of frames to be added for i in range(nb_frames): camera_visible_ids.time[0] = float(i) #time of the slice for j in range(X): for k in range(Y): camera_visible_ids.channel[0].detector[0].frame[0].image_raw[j,k] = float(j + k + i) #image_raw is a 2D array containing the data (pixels) of the frame if i==0: data_entry.put(camera_visible_ids) #the first frame has to be added using put() in order to store static data as well else: data_entry.put_slice(camera_visible_ids) #appending the slice (frame) to the IDS