All the steps are assumed to be performed at the EUROfusion gateway.
1. Connecting to databases
1.1. JET
Since 1st August 2021 connecting to JET data services via mdsplus.jetdata.eu requires SSH key authentication.
Below are instructions provided by JET codas team
In order to access the new mdsplus.jetdata.eu service, users will need to make a minor change to their codes, changing the host parameter
in the connection calls from "mdsplus.jet.efda.org" or "mdsplus.jet.uk" to "ssh://<username>@mdsplus.jetdata.eu"
where <username> is the your JET account username (e.g. jsmith or xy1234)
Users will also need to register their SSH public key with us in order for this to work.
Please do this by emailing support@it.ukaea.uk with the subject line “MDSplus SSH Key Registration”.
Please put your SSH Public key into the body of the message – do not add attachments to the email.
It is important that you provide your public key only and do not include your private key (users should take all reasonable steps to protect their private keys).
(Due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used)
Suggested email text to use:
Please register my SSH Public key for use with mdsplus.jetdata.eu.
My username is : <your shortname>
My SSH Public key is:
Once your key has been registered we will confirm this back to you and you will then be able to test the connection and your clients
If you need information about creating SSH keys please see the information here:
• There is some general information here: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-2
• There is a page on how to create key pairs with openssh at https://www.ssh.com/academy/ssh/keygen
• There is a page on how to create key pairs with putty at https://www.ssh.com/academy/ssh/putty/windows/puttygen
• The www.ssh.com has pages about generating keys for other ssh clients too.
Important Note: when generating SSH key pairs, we’d suggest that most users will find it more convenient not to use a passphrase (i.e. leave it blank).
Also, due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used
2. Saving data to IMAS
Before saving the experimental data to imas it is important check which version of imas data dictonary we are using.
You may load (or you may use scripts that load) IMAS environment without specifying the version
module purge
module load cineca
module load imasenv
The last line loads imas data dictionary and also idstools which will be used later.
However, loading imasenv does not necessarily provide you with the latest version of data dictionary. In this case:
~>module list
Currently Loaded Modulefiles:
1) profile/archive 18) cmake/3.5.2 35) IMAS/3.29.0/AL/4.8.3
2) cineca 19) mdsplus/7.92.0/gcc/4.8 36) itm-blas/3.8.0/intel/17.0
3) intel/pe-xe-2017--binary 20) blitz/1.0.1 37) itm-lapack/3.8.0/intel/17.0
4) itm-intel/17.0 21) jaxfront/R1.1 38) interpos/9.2.0/intel/17.0
5) intelmpi/2017--binary 22) git/2.23 39) xmllib/3.3.1/intel/17.0
6) itm-intelmpi/2017 23) itm-fftw/3.3.4 40) libfortranparser/0.0.6/intel/17.0
7) gnu/7.3.0 24) szip/2.1--gnu--6.1.0 41) keplertools/1.8.9
8) itm-gcc/7.3.0 25) zlib/1.2.8--gnu--6.1.0 42) kepler/2.5p5-3.1.1
9) jdk/1.8.0_111 26) itm-hdf5/1.8.17-old 43) imas-fc2k/4.13.9
10) itm-java/1.8.0_111 27) pspline/20161207 44) itm-qt/5.8.0
11) itm-python/3.6 28) slatec/4.1 45) imas-viz/2.4.4
12) matlab/2018b 29) itm-mkl/2017.1 46) idstools/1.5.1
13) itm-matlab/2018b 30) itm-matheval/1.1.11 47) autoGui/1.15
14) netbeans/7.3 31) itm-netcdf/4.4 48) ggd/1.9.1/intel/17.0/imas/3.29.0
15) itm-maven/3.3.9 32) nag/mark26--binary 49) libbds/1.0.2/intel/17.0/imas/3.29.0
16) scripts/R4.9 33) itm-nag/mark26--binary 50) amns/1.3.3/intel/17.0/imas/3.29.0
17) totalview/2017.3.8 34) uda/2.2.5 51) imasenv/3.29.0/intel/rc
we loaded version 3.29.0/intel/rc . If you want to load the latest version data dictionary check first what versions are available with a command "module avail <required module>":
~>module avail imasenv
---------------------------------------------------------- /gw/modules/environment ----------------------------------------------------------
imasenv/3.10.2 imasenv/3.23.1/ual/4.0.3/1.4 imasenv/3.26.0/gcc/7.3.0/rc imasenv/3.31.0/intel/17.0/1.0
imasenv/3.11.0 imasenv/3.23.1/ual/4.0.3/1.5 imasenv/3.26.0/intel/17.0/rc imasenv/3.31.0/intel/17.0/1.1
imasenv/3.12.1 imasenv/3.23.1/ual/4.0.3/1.6 imasenv/3.26.0/intel/rc imasenv/3.31.0/intel/17.0/rc
imasenv/3.15.1 imasenv/3.23.1/ual/4.0.4/1.0 imasenv/3.26.0/rc imasenv/3.31.0/intel/rc
imasenv/3.16.0 imasenv/3.23.1/ual/4.0.4/1.1 imasenv/3.28.0/gcc/7.3.0/rc imasenv/3.31.0/rc
imasenv/3.17.0 imasenv/3.23.1/ual/4.0.4/1.2 imasenv/3.28.0/intel/17.0/rc imasenv/3.32.0/gcc/7.3.0/rc
imasenv/3.17.1 imasenv/3.23.1/ual/4.1.0/1.0 imasenv/3.28.0/intel/rc imasenv/3.32.0/intel/17.0/rc
imasenv/3.18.0 imasenv/3.23.2/rc imasenv/3.28.0/rc imasenv/3.32.0/intel/rc
imasenv/3.19.0 imasenv/3.23.2/ual/4.1.1/1.0 imasenv/3.28.1/gcc/7.3.0/1.0 imasenv/3.32.0/rc
imasenv/3.19.1 imasenv/3.23.2/ual/4.1.2/0.2 imasenv/3.28.1/gcc/7.3.0/rc imasenv/3.32.1/gcc/7.3.0/rc
imasenv/3.20.0 imasenv/3.23.2/ual/4.1.2/1.0 imasenv/3.28.1/intel/17.0/1.0 imasenv/3.32.1/intel/17.0/rc
imasenv/3.21.0/ual/3.8.10/1.0 imasenv/3.23.2/ual/4.1.4/0.2 imasenv/3.28.1/intel/17.0/rc imasenv/3.32.1/intel/rc
imasenv/3.21.0/ual/3.8.5/1.0 imasenv/3.23.2/ual/4.1.4/1.0 imasenv/3.28.1/intel/rc imasenv/3.32.1/rc
imasenv/3.21.0/ual/3.8.8/1.0 imasenv/3.23.2/ual/4.1.5/1.0 imasenv/3.28.1/rc imasenv/3.33.0/gcc/7.3.0/rc
imasenv/3.21.1/ual/4.0.0/1.0 imasenv/3.23.2/ual/4.1.5/1.1 imasenv/3.29.0/gcc/7.3.0/1.0 imasenv/3.33.0/intel/17.0/rc
imasenv/3.21.1/ual/4.0.1/1.0 imasenv/3.23.2/ual/4.1.5/1.2 imasenv/3.29.0/gcc/7.3.0/rc imasenv/3.33.0/intel/rc
imasenv/3.22.0/ual/4.0.2/1.0 imasenv/3.24.0/rc imasenv/3.29.0/intel/17.0/1.0 imasenv/3.33.0/rc
imasenv/3.22.0/ual/4.0.2/1.1 imasenv/3.24.0/ual/4.1.5/1.0 imasenv/3.29.0/intel/17.0/rc imasenv/3.7.4
imasenv/3.22.0/ual/4.0.2/1.2 imasenv/3.24.0/ual/4.2.0/1.0 imasenv/3.29.0/intel/rc imasenv/3.8.0
imasenv/3.22.0/ual/4.0.2/1.3 imasenv/3.25.0/gcc/6.1.0/1.0 imasenv/3.29.0/rc imasenv/3.9.0
imasenv/3.22.0/ual/4.0.2/1.4 imasenv/3.25.0/gcc/6.1.0/rc imasenv/3.30.0/gcc/7.3.0/rc imasenv/3.9.1
imasenv/3.22.0/ual/4.0.2/1.5 imasenv/3.25.0/gcc/7.3.0/1.0 imasenv/3.30.0/intel/17.0/rc imasenv/newLL/3.18.0
imasenv/3.22.0/ual/4.0.2/1.6 imasenv/3.25.0/gcc/7.3.0/rc imasenv/3.30.0/intel/rc imasenv/test
imasenv/3.22.0/ual/4.0.2/1.7 imasenv/3.25.0/gcc/rc imasenv/3.30.0/rc imasenvX/3.19.1/ual/3.8.2/1.0
imasenv/3.22.0/ual/4.0.2/1.8 imasenv/3.25.0/intel/17.0/1.0 imasenv/3.31.0/1.0 imasenvX/3.20.0/ual/3.8.3/1.0
imasenv/3.23.1/ual/4.0.3/1.0 imasenv/3.25.0/intel/17.0/rc imasenv/3.31.0/1.1 imasenvX/3.20.0/ual/3.8.5/1.0
imasenv/3.23.1/ual/4.0.3/1.1 imasenv/3.25.0/intel/rc imasenv/3.31.0/gcc/7.3.0/1.0 imasenvX/3.21.0/ual/3.8.5/1.0
imasenv/3.23.1/ual/4.0.3/1.2 imasenv/3.25.0/rc imasenv/3.31.0/gcc/7.3.0/1.1
imasenv/3.23.1/ual/4.0.3/1.3 imasenv/3.25.0/ual/4.2.0/1.0 imasenv/3.31.0/gcc/7.3.0/rc
From the above output we conclude that the latest data dictionary version is 'imasenv/3.33.0/rc'. Using the above list you can also choose a different version if there is such a need.
Let's switch to the latest imas release available on the gateway:
module unload imasenv
module load imasenv/3.33.0/rc
You can now check if there exists ids to store the experimental data by typing:
Above command opens data dictionary by default in Konqueror. If option "Show/hide error bars" doesn't work, copying the address to Firefox will help.
By browsing data dictionaries you will find needed fiields. If there are no required fields then a request has to be raised.
![](/download/attachments/70879117/data_dict_top.png?version=1&modificationDate=1631786714411&api=v2)
![](/download/attachments/70879117/data_dict_bottom.png?version=1&modificationDate=1631786718594&api=v2)
2.1. Saving Thomson scattering data
In our example we want to save Thomson scattering data. You can find the corresponding ids in the list above. Let's browse this ids.
![](/download/attachments/70879117/data_dict_TS.png?version=1&modificationDate=1631787302343&api=v2)
We can imediately notice that there are structures introduced in imas version 3.32.1. Apart from experimentally measured values we can also save other experimental data like coordinates of line-of-sight of particular measurement.
Data is structured in such a way that we have to choose channel first to browse further for place to save measured values:
![](/download/attachments/70879117/data_dict_TS_te_ne.png?version=1&modificationDate=1631787305585&api=v2)
Let's assume we want save measured HRTS profiles of electron temperature and density with errors for the JET shot 99357.
After fetching data from JET we saved it in a dictionary with the following structure:
# hrts data with sig='TE','DTE','NE','DNE','Z'
# hrts[sig]={
# 'time': t, #1D float
# 'x':x, #1D float R coordinate
# 'data': data, #2D float except sig='Z'
# 'signal': signal,
# 'seq':sequence,
# 'uid':owner,
# 'dda':dda
# }
# 'data' is 2D data[i,j] where i=time_slice index and j= x_value index
'data' array consist of 701 time slices (from around t=40s to t=75s) and 63 experimental points. We need to project data onto numpy arrays:
import json
import imas,os,datetime,sys
import getpass
import numpy as np
from imas import imasdef
db = 'data_access_tutorial'
shot=99357
run=1
#creates the Data Entry object 'data_entry', a kind of handler of the pulse file with sho, run, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser())
#data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser())
# Open save data_entry to data base
# Tries to open data_entry
op = data_entry.open()
#if fails, creates one
if op[0]<0:
cp=data_entry.create()
if cp[0]==0:
print("data entry created")
elif op[0]==0:
print("data entry opened")
# Fetched data
with open("data/data_jet_hrts_99357.json") as json_file:
hrts=json.load(json_file)
x_coord=np.array(hrts['TE']['x'])
# no. of space points
nb_points = len(x_coord)
#no of time slices
nb_slices=len(hrts['TE']['time'])
#here, we can perform some read/write operations using the get/put() operations
#...
#creating the 'thomson_scattering' auxiliary IDS and initializing it
thomson = imas.thomson_scattering() #creates a 'thomson scattering' IDS
thomson.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)
thomson.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
#thomson.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime
# since all data is available we can save whole time vector at once
thomson.time=np.array(hrts['TE']['time'])
thomson.ids_properties.creation_date = datetime.datetime.now().strftime("%y-%m-%d")
thomson.channel.resize(nb_points)
for j in range(nb_points):
thomson.channel[j].t_e.data.resize(1)
thomson.channel[j].t_e.data_error_upper.resize(1)
thomson.channel[j].n_e.data.resize(1)
thomson.channel[j].n_e.data_error_upper.resize(1)
# python interface accepts only numpy arrays to be saved in ids
te_data = np.array(hrts['TE']['data']) #2D
dte_data = np.array(hrts['DTE']['data'])#2D
ne_data = np.array(hrts['NE']['data'])#2D
dne_data = np.array(hrts['DNE']['data'])#2D
z_data = np.array(hrts['Z']['data']) #1D
r_data = np.array(hrts['TE']['x']) #1D
for j in range(nb_points):
thomson.channel[j].position.r=r_data[j]
thomson.channel[j].position.z=z_data[j]
thomson.channel[j].t_e.data=te_data[:,j]
thomson.channel[j].t_e.data_error_upper=dte_data[:,j]
thomson.channel[j].n_e.data=ne_data[:,j]
thomson.channel[j].n_e.data_error_upper=dne_data[:,j]
data_entry.put(thomson)
Unknown User (michal.poradzinski@ifpilm.pl)