All the steps are assumed to be performed at the EUROfusion gateway.
1. Connecting to databases
1.1. JET
Since 1st August 2021 connecting to JET data services via mdsplus.jetdata.eu requires SSH key authentication.
Below are instructions provided by JET codas team
2. Saving data to IMAS
Before saving the experimental data to imas it is important check which version of imas data dictonary we are using.
You may load (or you may use scripts that load) IMAS environment without specifying the version
module purge module load cineca module load imasenv
The last line loads imas data dictionary and also idstools which will be used later.
However, loading imasenv does not necessarily provide you with the latest version of data dictionary. In this case:
we loaded version 3.29.0/intel/rc . If you want to load the latest version data dictionary check first what versions are available with a command "module avail <required module>":
From the above output we conclude that the latest data dictionary version is 'imasenv/3.33.0/rc'. Using the above list you can also choose a different version if there is such a need.
Let's switch to the latest imas release available on the gateway:
module unload imasenv module load imasenv/3.33.0/rc
You can now check if there exists ids to store the experimental data by typing:
dd_doc
Above command opens data dictionary by default in Konqueror. If option "Show/hide error bars" doesn't work, copying the address to Firefox will help.
By browsing data dictionaries you will find needed fiields. If there are no required fields then a request has to be raised.
2.1. Saving Thomson scattering data
In our example we want to save Thomson scattering data. You can find the corresponding ids in the list above. Let's browse this ids.
We can imediately notice that there are structures introduced in imas version 3.32.1. Apart from experimentally measured values we can also save other experimental data like coordinates of line-of-sight of particular measurement.
Data is structured in such a way that we have to choose channel first to browse further for place to save measured values:
Note: channel[i1].position.r and channel[i1].position.z are scalars.
Let's assume we want save measured HRTS profiles of electron temperature and density with errors for the JET shot 99357.
After fetching data from JET we saved it in a dictionary with the following structure:
# hrts data with sig='TE','DTE','NE','DNE','Z' # hrts[sig]={ # 'time': t, #1D float # 'x':x, #1D float R coordinate # 'data': data, #2D float except sig='Z' # 'signal': signal, # 'seq':sequence, # 'uid':owner, # 'dda':dda # } # 'data' is 2D data[i,j] where i=time_slice index and j= x_value index
'data' array consist of 701 time slices (from around t=40s to t=75s) and 63 experimental points. We need to project data onto numpy arrays:
import json import imas,os,datetime,sys import getpass import numpy as np from imas import imasdef db = 'data_access_tutorial' shot=99357 run=1 #creates the Data Entry object 'data_entry', a kind of handler of the pulse file with sho, run, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser()) #data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser()) # Open save data_entry to data base # Tries to open data_entry op = data_entry.open() #if fails, creates one if op[0]<0: cp=data_entry.create() if cp[0]==0: print("data entry created") elif op[0]==0: print("data entry opened") # Fetched data with open("data/data_jet_hrts_99357.json") as json_file: hrts=json.load(json_file) x_coord=np.array(hrts['TE']['x']) # no. of space points nb_points = len(x_coord) #no of time slices nb_slices=len(hrts['TE']['time']) #here, we can perform some read/write operations using the get/put() operations #... #creating the 'thomson_scattering' auxiliary IDS and initializing it thomson = imas.thomson_scattering() #creates a 'thomson scattering' IDS thomson.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory) thomson.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute #thomson.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime # since all data is available we can save whole time vector at once thomson.time=np.array(hrts['TE']['time']) thomson.ids_properties.creation_date = datetime.datetime.now().strftime("%y-%m-%d") thomson.channel.resize(nb_points) for j in range(nb_points): thomson.channel[j].t_e.data.resize(1) thomson.channel[j].t_e.data_error_upper.resize(1) thomson.channel[j].n_e.data.resize(1) thomson.channel[j].n_e.data_error_upper.resize(1) # python interface accepts only numpy arrays to be saved in ids te_data = np.array(hrts['TE']['data']) #2D dte_data = np.array(hrts['DTE']['data'])#2D ne_data = np.array(hrts['NE']['data'])#2D dne_data = np.array(hrts['DNE']['data'])#2D z_data = np.array(hrts['Z']['data']) #1D r_data = np.array(hrts['TE']['x']) #1D for j in range(nb_points): thomson.channel[j].position.r=r_data[j] thomson.channel[j].position.z=z_data[j] thomson.channel[j].t_e.data=te_data[:,j] thomson.channel[j].t_e.data_error_upper=dte_data[:,j] thomson.channel[j].n_e.data=ne_data[:,j] thomson.channel[j].n_e.data_error_upper=dne_data[:,j] data_entry.put(thomson)
Note that we used only one command 'data_entry.put(thomson)' . Since we had all data available at once we didn't need to use put and putSlice commands to save slice by slice. However, the two approaches should be equivalent and provide the same ids. The first one is faster as it saving is done only once.
If the data was saved with MDSPLUS backend , the shot number is included in the data files names and is saved in the following folder:
> ~/public/imasdb/data_access_tutorial/3/0>ls ids_150000001.characteristics ids_150000001.tree ids_993570001.datafile ids_150000001.datafile ids_993570001.characteristics ids_993570001.tree
If the data is saved using HDF5 backend, the shot number is included in the data tree folder names and is saved in the following folder:
> ~/public/imasdb/data_access_tutorial/3/99357/1>ls master.h5 thomson_scattering.h5
If you used HDF5 backend you can check if the data was stored correctly in the ids directly with the command:
>~/public/imasdb/data_access_tutorial/3/99357/1>h5dump thomson_scattering.h5 |less
2.2. Reading data
Once the data is saved to ids we can open it and for example plot it:
import imas import numpy as np import matplotlib.pyplot as plt import sys from imas import imasdef db = 'data_access_tutorial' shot=99357 run=1 #creating the Data Entry object which handles the pulse file with shot=15000, run=1, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend #data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser()) data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser()) #opening the pulse file handled by the Data Entry object previously created data_entry.open() #getting a slice at time=1s using the closest time slice interpolation time_requested=50. sslice = data_entry.get_slice('thomson_scattering', time_requested, imasdef.CLOSEST_INTERP) no_channels=len(sslice.channel) #[te,dte,ne,dne,r,z] =[np.array([0.0]*no_channels)]*6 te=np.array([]) dte=np.array([]) ne=np.array([]) dne=np.array([]) r=np.array([]) for i in range(no_channels): te=np.append(te,sslice.channel[i].t_e.data) dte= np.append(dte,sslice.channel[i].t_e.data_error_upper) ne=np.append(ne,sslice.channel[i].n_e.data) dne= np.append(dne,sslice.channel[i].n_e.data_error_upper) r= np.append(r,sslice.channel[i].position.r) z[i]= sslice.channel[i].position.z #closing the Data Entry data_entry.close() # plotting fig1=plt.figure(1) elinewidth=0.9 capsize=3 ax1=fig1.add_subplot(121) ax1.errorbar(r,te,yerr=dte,fmt='o',elinewidth=elinewidth,capsize=capsize, color='blue',label=r'$T_e$ at $t='+str(time_requested)+'$ s') ax1.set_ylim([0.,6.0e3]) ax1.set_xlabel("r (m)",fontsize=14) ax1.set_ylabel("eV",fontsize=14) ax1.legend(loc='lower left', bbox_to_anchor=(0.0, 0.0),fontsize=12) ax2=fig1.add_subplot(122) ax2.errorbar(r,ne,yerr=dne,fmt='o',elinewidth=elinewidth,capsize=capsize, color='red',label=r'$n_e$ at $t='+str(time_requested)+'$ s') ax2.set_ylim([0.,7.2e19]) ax2.set_xlabel("r (m)",fontsize=14) ax2.set_ylabel("$m^{-3}$",fontsize=14) ax2.legend(loc='lower left', bbox_to_anchor=(0.0, 0.0),fontsize=12) plt.show()
As it can be seen from the plot, the fetched data has not been yet filtered.