Code Block

title	Connecting to JET - instructions
collapse	true

In order to access the new mdsplus.jetdata.eu service, users will need to make a minor change to their codes, changing the host parameter 
in the connection calls from "mdsplus.jet.efda.org" or "mdsplus.jet.uk" to "ssh://<username>@mdsplus.jetdata.eu" 
where <username> is the your JET account username (e.g. jsmith or xy1234)

Users will also need to register their SSH public key with us in order for this to work. 
Please do this by emailing support@it.ukaea.uk with the subject line “MDSplus SSH Key Registration”. 
Please put your SSH Public key into the body of the message – do not add attachments to the email. 
It is important that you provide your public key only and do not include your private key (users should take all reasonable steps to protect their private keys). 
(Due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used)

Suggested email text to use:

Please register my SSH Public key for use with mdsplus.jetdata.eu. 
My username is : <your shortname>
My SSH Public key is: 

Once your key has been registered we will confirm this back to you and you will then be able to test the connection and your clients

If you need information about creating SSH keys please see the information here:
•	There is some general information here: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-2
•	There is a page on how to create key pairs with openssh at https://www.ssh.com/academy/ssh/keygen
•	There is a page on how to create key pairs with putty at https://www.ssh.com/academy/ssh/putty/windows/puttygen
•	The www.ssh.com has pages about generating keys for other ssh clients too.

Important Note: when generating SSH key pairs, we’d suggest that most users will find it more convenient not to use a passphrase (i.e. leave it blank). 
Also, due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used

Saving data to IMAS

Before saving the experimental data to imas it is important check which version of imas data dictonary we are using.

You may load (or you may use scripts that load) IMAS environment without specifying the version

Code Block
module purge module load cineca module load imasenv

The last line loads imas data dictionary and also idstools which will be used later.

However, loading imasenv does not necessarily provide you with the latest version of data dictionary. In this case:

Once the pair of keys 'id_rsa_jet' is created it necessary that the system can distinguish between the keys. You can inform the system by editing 'config' file in ~/ .ssh folder.

Code Block

title	ssh config

Host jet
 HostName mdsplus.jetdata.eu
 User <your jet username>
 IdentityFile ~/.ssh/id_rsa_jet

In an example we fetch JET data (dda='hrts', uid = 'jetppf', seq=0, dtype= ['TE', 'DTE', 'NE', 'DNE', 'Z']) .

Example of script fetching HRTS JET data.

Code Block

title	Example script of fetching data from JET
linenumbers	true
collapse	true

#example of saving experimental data to ids

import os
import json	

def read_ppf(conn, shot, ppf, seq=0, uid='jetppf',debug=False):
    ierr = conn.get('_sig=ppfuid("' + uid +'")')
    c = '_sig=jet("ppf/%s/%d",%d)'% (ppf, seq, shot)
    if debug: print('\nDEBUG: %s\n' % c)
    try:

Code Block

language	bash
title	>module list
collapse	true

~>module list
Currently Loaded Modulefiles:
  1) profile/archive                      18) cmake/3.5.2                          35) IMAS/3.29.0/AL/4.8.3
  2) cineca         s = conn.get(c)
        raw = s.data()
        dim0  19) mdsplus/7.92.0/gcc/4.8= conn.get('dim_of(_sig,0)').data()
        try:
         36) itm-blas/3.8.0/intel/17.0
  3) intel/pe-xe-2017--binary dim1 = conn.get('dim_of(_sig,1)').data()
          return {'raw': raw, 'x':dim0, 't':dim1}
  20) blitz/1.0.1     except:
          return {'raw': raw, 't':dim0}
    except:
    37) itm-lapack/3.8.0/intel/17.0
  4) itm-intel/17.0  return None


def readExperimental(data_exp, host, server='ssh://'):
    '''
    reads experimental data after connecting to the host via ssh server
     21) jaxfront/R1.1   data_exp: data details in dictionary format	
    '''
    import sys
    try:	
        38) interpos/9.2.0/intel/17.0import MDSplus
  5) intelmpi/2017--binary     haveMDS = True
    except:
     22) git/2.23  print(' No MDSplus support found.\n')
        print('\n\n\n Exiting...\n')
        exit(1)

    try:
  39) xmllib/3.3.1/intel/17.0
  6) itm-intelmpi/2017   conn = MDSplus.Connection(server+host)
      print('Connection OK')
      connected = True
   23) itm-fftw/3.3.4 except:
      print('in readExperimental, MDSplus failed: ',sys.exc_info()[1])
      connected = False

     40) libfortranparser/0.0.6/intel/17.0if connected:
  7) gnu/7.3.0     try:
            EXP={}
            for sig  24) szip/2.1--gnu--6.1.0in data_exp['dtype']:
                print('reading dtype: 41',sig) keplertools/1.8.9

    8) itm-gcc/7.3.0           signal= data_exp['dda']+'/'+sig
            25) zlib/1.2.8--gnu--6.1.0            aux = read_ppf(conn, discharge, signal, seq=data_exp['seq'], uid=data_exp['uid']) 
       42) kepler/2.5p5-3.1.1
  9) jdk/1.8.0_111     # Z coordinate is one dimensional, aux['raw'] is 2D by default
         26) itm-hdf5/1.8.17-old      if sig=='Z':
           43) imas-fc2k/4.13.9
 10) itm-java/1.8.0_111      aux['raw'] = aux['raw'].flatten()

           27) pspline/20161207    EXP[sig] =  {
              44) itm-qt/5.8.0
 11) itm-python/3.6   'time': aux['t'].tolist(),
                    28) slatec/4.1'x':aux['x'].tolist(),
                    'data': aux['raw'].tolist(),
        45) imas-viz/2.4.4
 12) matlab/2018b         'signal': signal 
              29) itm-mkl/2017.1     }
        except:
          46) idstools/1.5.1
 13) itm-matlab/2018b    print('in readExperimental, read_ppf failed: ',sys.exc_info()[1])
                  30) itm-matheval/1.1.11return None
                   47) autoGui/1.15
 14) netbeans/7.3del(conn)
    else:
        EXP=None
    
    return EXP

###########


host='jet'

discharge=99357

# data to download
data={}
data['dda']='hrts'
data['uid'] = 'jetppf'
data['seq']=0
data['dtype']=['TE','DTE','NE','DNE','Z']

EXP=readExperimental(data,host)

if EXP:
    a_file = open('data_jet_fetched_'+str(discharge)+'_'+data['dda']+'_'+str(data['seq'])+'.json', "w")
    json.dump(EXP, a_file)
    a_file.close()

IMAS UDA

It is possible to access and fetch and map data from several experiments WEST, JET , TCV and AUG using UDA protocol which id described in detailed on ITER confulence pages (requires iter account). Accessing data with UDA needs to be adapted due to recent (Aug 2021) change in connection protocol.

Saving data to IMAS

Before saving the experimental data to imas it is important check which version of imas data dictonary we are using.

You may load (or you may use scripts that load) IMAS environment without specifying the version

Code Block
module purge module load cineca module load imasenv

The last line loads imas data dictionary and also idstools which will be used later.

However, loading imasenv does not necessarily provide you with the latest version of data dictionary. In this case:

Code Block

language	bash
title	>module list
collapse	true

~>module list
Currently Loaded Modulefiles:
  1) profile/archive    31) itm-netcdf/4.4                       48) ggd/1.9.1/intel/17.0/imas/3.29.0
 15) itm-maven/3.3.9                      32) nag/mark26--binary                   49) libbds/1.0.2/intel/17.0/imas/3.29.0
 16) scripts/R4.9                         33) itm-nag/mark26--binary               50) amns/1.3.3/intel/17.0/imas/3.29.0
 17) totalview/2017.3.8                   3418) udacmake/23.25.52                            5135) imasenvIMAS/3.29.0/intel/rc

we loaded version 3.29.0/intel/rc . If you want to load the latest version data dictionary check first what versions are available with a command "module avail <required module>":

Code Block

language	bash
title	>module avail imasenv
collapse	true

~>module avail imasenv

---------------------------------------------------------- /gw/modules/environment ----------------------------------------------------------
imasenv/3.10.2AL/4.8.3
  2) cineca                               19) mdsplus/7.92.0/gcc/4.8               36) itm-blas/3.8.0/intel/17.0
  3) intel/pe-xe-2017--binary             20) blitz/1.0.1                          37) itm-lapack/3.8.0/intel/17.0
  4) itm-intel/17.0                       21) imasenvjaxfront/3.23.1/ual/4.0.3/1.4  imasenv/3.26.0/gcc/7.3.0/rc   imasenv/3.31R1.1                        38) interpos/9.2.0/intel/17.0/1.0
imasenv/3.11.0  5) intelmpi/2017--binary                22) imasenvgit/3.23.1/ual/4.0.3/1.5  imasenv/3.26.0/intel/17.0/rc  imasenv/3.31.0/intel/17.0/1.1
imasenv/3.12.12.23                             39) imasenvxmllib/3.233.1/ualintel/417.0.3/1.6  imasenv/3.26.0/intel/rc
  6) itm-intelmpi/2017       imasenv/3.31.0/intel/17.0/rc
imasenv/3.15.1                imasenv23) itm-fftw/3.233.1/ual/4.0.4/1.0  imasenv/3.26.0/rc4                       40) imasenvlibfortranparser/30.310.06/intel/rc
imasenv/3.16.017.0
  7) gnu/7.3.0                            24) imasenvszip/3.23.1/ual/4.0.4/1.1  imasenv/3.28.0/2.1--gnu--6.1.0                 41) keplertools/1.8.9
  8) itm-gcc/7.3.0/rc   imasenv/3.31.0/rc
imasenv/3.17                        25) zlib/1.2.8--gnu--6.1.0               42) imasenvkepler/2.5p5-3.231.1/ual/4.0.4/1.2  imasenv/3.28.0/intel/17.0/rc  imasenv/3.32.0/gcc/7.3.0/rc
imasenv/3.17.1
  9) jdk/1.8.0_111                imasenv/3.23.1/ual/4.1.0/1.0  imasenv/3.28.0/intel/rc      26) imasenvitm-hdf5/31.328.0/intel/17.0/rc
imasenv/3.18.017-old                  43) imasenvimas-fc2k/34.23.2/rc13.9
 10) itm-java/1.8.0_111             imasenv/3.28.0/rc      27) pspline/20161207      imasenv/3.32.0/intel/rc
imasenv/3.19.0               44) imasenvitm-qt/35.23.2/ual/4.1.1/1.0  imasenv/3.28.1/gcc/7.3.0/1.0  imasenv/3.32.0/rc
imasenv/3.19.18.0
 11) itm-python/3.6                        imasenv/3.23.2/ual28) slatec/4.1.2/0.2  imasenv/3.28.1/gcc/7.3.0/rc   imasenv/3.32.1/gcc/7.3.0/rc
imasenv/3.20.0                           45) imas-viz/2.4.4
 12) matlab/2018b                         29) imasenv/3.23.2/ual/4.1.2/1.0  imasenv/3.28.1/intel/17.0/1.0 imasenv/3.32.1/intel/17.0/rc
imasenv/3.21.0/ual/3.8.10/1.0 imasenv/3.23.2/ual/4.1.4/0.2  imasenv/3.28.1/intel/17.0/rc  imasenv/3.32.1/intel/rc
imasenv/3.21.0/ual/3.8.5/1.0  imasenv/3.23.2/ual/4.1.4/1.0  imasenv/3.28.1/intel/rcitm-mkl/2017.1                       46) idstools/1.5.1
 13) itm-matlab/2018b                     30) itm-matheval/1.1.11                  imasenv/3.32.1/rc
imasenv/3.21.0/ual/3.8.8/1.0  imasenv/3.23.2/ual/4.1.5/1.0  imasenv/3.28.1/rc47) autoGui/1.15
 14) netbeans/7.3                         31) itm-netcdf/4.4                  imasenv/3.33.0/gcc/7.3.0/rc
imasenv/3.21.1/ual/4.0.0/1.0  imasenv/3.23.2/ual/4.1.5/1.1  imasenv/3.29.0/gcc/7.3.0/1.0  imasenv/3.33.0/intel/17.0/rc
imasenv/3.21.1/ual/4.0.1/1.0  imasenv/3.23.2/ual/4.1.5/1.2  imasenv/3.29.0/gcc/7.3.0/rc   imasenv/3.33.0/intel/rc
imasenv/3.22.0/ual/4.0.2/1.0  imasenv/3.24.0/rc     48) ggd/1.9.1/intel/17.0/imas/3.29.0
 15) itm-maven/3.3.9                      32) imasenv/3.29.0nag/mark26--binary                   49) libbds/1.0.2/intel/17.0/1.0 imasenvimas/3.3329.0/rc
imasenv/3.22.0/ual/4.0.2/1.1  imasenv/3.24.0/ual/4.1.5/1.0  imasenv/3.29.0/intel/17.0/rc  imasenv/3.7.4
imasenv/3.22.0/ual/4.0.2/1.2  imasenv/3.24.0/ual/4.2.0/1.0  imasenv/3.29.0/intel/rc
 16) scripts/R4.9                         33) itm-nag/mark26--binary               50) amns/1.3.3/intel/17.0/imas/3.29.0
 17) totalview/2017.3.8                   34) imasenv/3.8.0
imasenv/3.22.0/ual/4.0.2/1.3  imasenv/3.25.0/gcc/6.1.0/1.0  imasenv/3.29.0/rcuda/2.2.5                            51) imasenv/3.929.0
imasenv//intel/rc

we loaded version 3.

...

29.0/

...

intel/rc . If you want to load the latest version data dictionary check first what versions are available with a command "module avail <required module>":

Code Block

language	bash
title	>module avail imasenv
collapse	true

~>module avail imasenv

---------------------------------------------------------- /gw/modules/environment ----------------------------------------------------------
imasenv/3.10.2                imasenv/3.23.1/ual/4.0.3/1.44.0.2/1.4  imasenv/3.25.0/gcc/6.1.0/rc   imasenv/3.30.0/gcc/7.3.0/rc   imasenv/3.9.1
imasenv/3.22.0/ual/4.0.2/1.5  imasenv/3.25.0/gcc/7.3.0/1.0  imasenv/3.3026.0/intelgcc/177.3.0/rc   imasenv/newLL/3.18/3.31.0/intel/17.0/1.0
imasenv/3.2211.0                imasenv/3.23.1/ual/4.0.23/1.65  imasenv/3.2526.0/gccintel/7.317.0/rc   imasenv/3.3031.0/intel/rc/17.0/1.1
imasenv/3.12.1                imasenv/test
imasenv/3.2223.01/ual/4.0.23/1.76  imasenv/3.2526.0/gccintel/rc         imasenv/3.30.31.0/intel/17.0/rc
imasenv/3.15.1                imasenvXimasenv/3.1923.1/ual/34.80.24/1.0
  imasenv/3.2226.0/ual/4.0.2/1.8rc             imasenv/3.2531.0/intel/17.0/1.0 rc
imasenv/3.3116.0/1.0                imasenvXimasenv/3.2023.01/ual/34.80.34/1.0
1  imasenv/3.2328.10/ualgcc/4.07.3/1.0/rc   imasenv/3.25.0/intel/1731.0/rc  
imasenv/3.3117.0/1.1                imasenvX/3.20.0/ual/3.8.5/1.0
imasenv/3.23.1/ual/4.0.34/1.12  imasenv/3.2528.0/intel/17.0/rc       imasenv/3.3132.0/gcc/7.3.0/1.0  imasenvXrc
imasenv/3.21.0/ual/3.8.5/1.0
17.1                imasenv/3.23.1/ual/4.1.0.3/1.20  imasenv/3.2528.0/intel/rc             imasenv/3.3132.0/gccintel/7.317.0/1.1rc
imasenv/3.23.1/ual/4.0.3/1.318.0                imasenv/3.25.0/ual/423.2.0/1.0/rc             imasenv/3.3128.0/gcc/7.3.0/rc

...

rc             imasenv/3.32.0/intel/rc
imasenv/3.

...

19.0

...

Let's switch to the latest imas release available on the gateway:

Code Block
module unload imasenv module load imasenv/3.23.2/ual/4.1.1/1.0 imasenv/3.28.1/gcc/7.3.0/1.0 imasenv/3.3332.0/rc

You can now check if there exists ids to store the experimental data by typing:

Code Block
dd_doc

Above command opens data dictionary by default in Konqueror. If option "Show/hide error bars" doesn't work, copying the address to Firefox will help.

By browsing data dictionaries you will find needed fiields. If there are no required fields then a request has to be raised.

Image Removed

Saving Thomson scattering data

In our example we want to save Thomson scattering data. You can find the corresponding ids in the list above. Let's browse this ids.

Image Removed

We can imediately notice that there are structures introduced in imas version 3.32.1. Apart from experimentally measured values we can also save other experimental data like coordinates of line-of-sight of particular measurement.

Data is structured in such a way that we have to choose channel first to browse further for place to save measured values:

Image Removed

Note: channel[i1].position.r and channel[i1].position.z are scalars.

Let's assume we want save measured HRTS profiles of electron temperature and density with errors for the JET shot 99357.

After fetching data from JET we saved it in a dictionary with the following structure:

Code Block

title	Fetched HRTS data

# hrts data with sig='TE','DTE','NE','DNE','Z' 
#    hrts[sig]={
#        'time': t, #1D float
#        'x':x, #1D float R coordinate
#        'data': data, #2D float except sig='Z'
#        'signal': signal,
#        'seq':sequence,
#        'uid':owner,
#        'dda':dda
#               }
# 'data' is 2D data[i,j] where i=time_slice index and j= x_value index

'data' array consist of 701 time slices (from around t=40s to t=75s) and 63 experimental points. We need to project data onto numpy arrays:


imasenv/3.19.1                imasenv/3.23.2/ual/4.1.2/0.2  imasenv/3.28.1/gcc/7.3.0/rc   imasenv/3.32.1/gcc/7.3.0/rc
imasenv/3.20.0                imasenv/3.23.2/ual/4.1.2/1.0  imasenv/3.28.1/intel/17.0/1.0 imasenv/3.32.1/intel/17.0/rc
imasenv/3.21.0/ual/3.8.10/1.0 imasenv/3.23.2/ual/4.1.4/0.2  imasenv/3.28.1/intel/17.0/rc  imasenv/3.32.1/intel/rc
imasenv/3.21.0/ual/3.8.5/1.0  imasenv/3.23.2/ual/4.1.4/1.0  imasenv/3.28.1/intel/rc       imasenv/3.32.1/rc
imasenv/3.21.0/ual/3.8.8/1.0  imasenv/3.23.2/ual/4.1.5/1.0  imasenv/3.28.1/rc             imasenv/3.33.0/gcc/7.3.0/rc
imasenv/3.21.1/ual/4.0.0/1.0  imasenv/3.23.2/ual/4.1.5/1.1  imasenv/3.29.0/gcc/7.3.0/1.0  imasenv/3.33.0/intel/17.0/rc
imasenv/3.21.1/ual/4.0.1/1.0  imasenv/3.23.2/ual/4.1.5/1.2  imasenv/3.29.0/gcc/7.3.0/rc   imasenv/3.33.0/intel/rc
imasenv/3.22.0/ual/4.0.2/1.0  imasenv/3.24.0/rc             imasenv/3.29.0/intel/17.0/1.0 imasenv/3.33.0/rc
imasenv/3.22.0/ual/4.0.2/1.1  imasenv/3.24.0/ual/4.1.5/1.0  imasenv/3.29.0/intel/17.0/rc  imasenv/3.7.4
imasenv/3.22.0/ual/4.0.2/1.2  imasenv/3.24.0/ual/4.2.0/1.0  imasenv/3.29.0/intel/rc       imasenv/3.8.0
imasenv/3.22.0/ual/4.0.2/1.3  imasenv/3.25.0/gcc/6.1.0/1.0  imasenv/3.29.0/rc             imasenv/3.9.0
imasenv/3.22.0/ual/4.0.2/1.4  imasenv/3.25.0/gcc/6.1.0/rc   imasenv/3.30.0/gcc/7.3.0/rc   imasenv/3.9.1
imasenv/3.22.0/ual/4.0.2/1.5  imasenv/3.25.0/gcc/7.3.0/1.0  imasenv/3.30.0/intel/17.0/rc  imasenv/newLL/3.18.0
imasenv/3.22.0/ual/4.0.2/1.6  imasenv/3.25.0/gcc/7.3.0/rc   imasenv/3.30.0/intel/rc       imasenv/test
imasenv/3.22.0/ual/4.0.2/1.7  imasenv/3.25.0/gcc/rc         imasenv/3.30.0/rc             imasenvX/3.19.1/ual/3.8.2/1.0
imasenv/3.22.0/ual/4.0.2/1.8  imasenv/3.25.0/intel/17.0/1.0 imasenv/3.31.0/1.0            imasenvX/3.20.0/ual/3.8.3/1.0
imasenv/3.23.1/ual/4.0.3/1.0  imasenv/3.25.0/intel/17.0/rc  imasenv/3.31.0/1.1            imasenvX/3.20.0/ual/3.8.5/1.0
imasenv/3.23.1/ual/4.0.3/1.1  imasenv/3.25.0/intel/rc       imasenv/3.31.0/gcc/7.3.0/1.0  imasenvX/3.21.0/ual/3.8.5/1.0
imasenv/3.23.1/ual/4.0.3/1.2  imasenv/3.25.0/rc             imasenv/3.31.0/gcc/7.3.0/1.1
imasenv/3.23.1/ual/4.0.3/1.3  imasenv/3.25.0/ual/4.2.0/1.0  imasenv/3.31.0/gcc/7.3.0/rc

From the above output we conclude that the latest data dictionary version is 'imasenv/3.33.0/rc'. Using the above list you can also choose a different version if there is such a need.

Let's switch to the latest imas release available on the gateway:

Code Block
module unload imasenv module load imasenv/3.33.0/rc

You can now check if there exists ids to store the experimental data by typing:

Code Block
dd_doc

Above command opens data dictionary by default in Konqueror. If option "Show/hide error bars" doesn't work, copying the address to Firefox will help.

By browsing data dictionaries you will find needed fiields. If there are no required fields then a request has to be raised.

Image Added

Saving Thomson scattering data

In our example we want to save Thomson scattering data. You can find the corresponding ids in the list above. Let's browse this ids.

Image Added

We can imediately notice that there are structures introduced in imas version 3.32.1. Apart from experimentally measured values we can also save other experimental data like coordinates of line-of-sight of particular measurement.

Data is structured in such a way that we have to choose channel first to browse further for place to save measured values:

Image Added

Note: channel[i1].position.r and channel[i1].position.z are scalars.

Let's assume we want to save measured HRTS profiles of electron temperature and density with errors for the JET shot 99357.

After fetching data from JET (dda='hrts', uid = 'jetppf', seq=0, dtype= ['TE', 'DTE', 'NE', 'DNE', 'Z']) we saved it in a dictionary with the following structure:

Code Block

title	Fetched HRTS data

# hrts data with sig='TE','DTE','NE','DNE','Z' 
#    hrts[sig]={
#        'time': t, #1D float
#        'x':x, #1D float R coordinate
#        'data': data, #2D float except sig='Z'
#        'signal': signal,
#        'seq':sequence,
#        'uid':owner,
#        'dda':dda
#               }
# 'data' is 2D data[i,j] where i=time_slice index and j= x_value index

'data' array consist of 701 time slices (from around t=40s to t=75s) and 63 experimental points.

Code Block

title	saving to imas

import json
import imas,os,datetime,sys
import getpass
import numpy as np
from imas import imasdef
 

db = 'data_access_tutorial'
shot=99357 
run=1

#creates the Data Entry object 'data_entry', a kind of handler of the pulse file with sho, run, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser())
#data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser())
# Open save data_entry to data base

# Tries to open data_entry
op = data_entry.open()
#open() and create(0 return a tuple (x_int,y_int) where x<0, y>0, x number of failures, y number of successes in the current session. 
#if open fails, create data_entry
if op[0]<0:
     cp=data_entry.create()
     if cp[0]==0:
        print("data entry created")
elif op[0]==0:
    print("data entry opened")	

# Open file with fetched data, the data is not numpy array format
with open("data/data_jet_hrts_99357.json") as json_file:
    hrts=json.load(json_file)

x_coord=np.array(hrts['TE']['x'])
# no. of space points
nb_points = len(x_coord)
#no of time slices
nb_slices=len(hrts['TE']['time'])

#creating the 'thomson_scattering' auxiliary IDS and initializing it
thomson = imas.thomson_scattering() #creates a 'thomson scattering' IDS
thomson.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)	
thomson.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
#thomson.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime

# since all data is available we can save whole time vector at once
thomson.time=np.array(hrts['TE']['time'])

thomson.ids_properties.creation_date = datetime.datetime.now().strftime("%y-%m-%d")
# the number of channel corresponds to number of data points
thomson.channel.resize(nb_points)

for j in range(nb_points):
    thomson.channel[j].t_e.data.resize(1)
    thomson.channel[j].t_e.data_error_upper.resize(1)
    thomson.channel[j].n_e.data.resize(1)
    thomson.channel[j].n_e.data_error_upper.resize(1)

# python interface accepts only numpy arrays to be saved in ids
te_data = np.array(hrts['TE']['data']) #2D
dte_data = np.array(hrts['DTE']['data'])#2D
ne_data = np.array(hrts['NE']['data'])#2D
dne_data = np.array(hrts['DNE']['data'])#2D	
z_data = np.array(hrts['Z']['data']) #1D
r_data = np.array(hrts['TE']['x']) #1D
	
for j in range(nb_points):
    thomson.channel[j].position.r=r_data[j]
    thomson.channel[j].position.z=z_data[j] 
    thomson.channel[j].t_e.data=te_data[:,j]
    thomson.channel[j].t_e.data_error_upper=dte_data[:,j]
    thomson.channel[j].n_e.data=ne_data[:,j]
    thomson.channel[j].n_e.data_error_upper=dne_data[:,j]

sequence = hrts['TE']['seq'] 
data_entry.put(thomson,seq)# the last number is the occurence which can be used to store the data sequence number
#closing the Data Entry
data_entry.close()

Note, that we used (once and outside the loop) only one command 'data_entry.put(thomson)' to save the data. Since we had all data available at once we didn't need to use put and putSlice commands to save time slice by time slice. However, the two approaches should be equivalent and provide the same ids. The first one is faster as procedure of saving the data in the physical memory is performed only once.

If the data was saved with MDSPLUS backend , the shot number is included in the data files names and is saved in the following folder:

Code Block
> ~/public/imasdb/data_access_tutorial/3/0>ls ids_150000001.characteristics ids_150000001.tree ids_993570001.datafile ids_150000001.datafile ids_993570001.characteristics ids_993570001.tree

If the data is saved using HDF5 backend, the shot number is included in the data tree folder names and is saved in the following folder:

Code Block
> ~/public/imasdb/data_access_tutorial/3/99357/1>ls master.h5 thomson_scattering.h5

If you used HDF5 backend you can check if the data was stored correctly in the ids directly with the command:

Code Block
>~/public/imasdb/data_access_tutorial/3/99357/1>h5dump thomson_scattering.h5 \|less

Code Block

title	h5dump output
collapse	true

HDF5 "thomson_scattering.h5" {
GROUP "/" {
   ATTRIBUTE "HDF5_BACKEND_VERSION" {
      DATATYPE  H5T_STRING {
         STRSIZE 10;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "1.0"
      }
   }
   ATTRIBUTE "RUN" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 1
      }
   }
   ATTRIBUTE "SHOT" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 99357
      }
   }
   GROUP "thomson_scattering" {
      DATASET "channel[]&AOS_SHAPE" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 1 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 63
         }
      }
      DATASET "channel[]&n_e&data" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 63, 701 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
         DATA {
         (0,0): 1e-20, 1e-20, 1.74005e+18, 3.30191e+18, 4.63574e+18,
         (0,5): 5.01502e+18, 7.16228e+18, 9.61972e+18, 1.14838e+19,
         (0,9): 1.26156e+19, 1.40738e+19, 1.49135e+19, 1.7819e+19,
         (0,13): 1.69414e+19, 1.83607e+19, 1.80031e+19, 1.95752e+19,
         (0,17): 1.92397e+19, 1.74838e+19, 1.87457e+19, 1.84743e+19,
         (0,21): 1.99327e+19, 1.93234e+19, 1.93735e+19, 2.02663e+19,
         (0,25): 1.82691e+19, 1.79814e+19, 1.55414e+19, 1.5825e+19,
         (0,29): 1.47801e+19, 1.63837e+19, 1.40053e+19, 1.41187e+19,
         (0,33): 1.46383e+19, 1.25988e+19, 1.40001e+19, 1.20421e+19,
         (0,37): 1.31313e+19, 1.51261e+19, 1.39026e+19, 1.41953e+19,
         (0,41): 1.52628e+19, 1.4697e+19, 1.56486e+19, 1.55934e+19,
         (0,45): 1.79571e+19, 1.52815e+19, 1.73831e+19, 1.69241e+19,
         (0,49): 1.80319e+19, 1.77864e+19, 1.8345e+19, 1.91198e+19,
         (0,53): 1.7536e+19, 2.03026e+19, 1.91662e+19, 1.97445e+19,
:

Code Block

title	saving to imas

import json
import imas,os,datetime,sys
import getpass
import numpy as np
from imas import imasdef
 

db = 'data_access_tutorial'
shot=99357 
run=1

#creates the Data Entry object 'data_entry', a kind of handler of the pulse file with sho, run, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend
data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser())
#data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser())
# Open save data_entry to data base

# Tries to open data_entry
op = data_entry.open()
#if fails, creates one
if op[0]<0:
     cp=data_entry.create()
     if cp[0]==0:
        print("data entry created")
elif op[0]==0:
    print("data entry opened")	

# Fetched data
with open("data/data_jet_hrts_99357.json") as json_file:
    hrts=json.load(json_file)

x_coord=np.array(hrts['TE']['x'])
# no. of space points
nb_points = len(x_coord)
#no of time slices
nb_slices=len(hrts['TE']['time'])

#here, we can perform some read/write operations using the get/put() operations
#...

#creating the 'thomson_scattering' auxiliary IDS and initializing it
thomson = imas.thomson_scattering() #creates a 'thomson scattering' IDS
thomson.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory)	
thomson.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute
#thomson.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime

# since all data is available we can save whole time vector at once
thomson.time=np.array(hrts['TE']['time'])

thomson.ids_properties.creation_date = datetime.datetime.now().strftime("%y-%m-%d")

thomson.channel.resize(nb_points)
for j in range(nb_points):
    thomson.channel[j].t_e.data.resize(1)
    thomson.channel[j].t_e.data_error_upper.resize(1)
    thomson.channel[j].n_e.data.resize(1)
    thomson.channel[j].n_e.data_error_upper.resize(1)

# python interface accepts only numpy arrays to be saved in ids
te_data = np.array(hrts['TE']['data']) #2D
dte_data = np.array(hrts['DTE']['data'])#2D
ne_data = np.array(hrts['NE']['data'])#2D
dne_data = np.array(hrts['DNE']['data'])#2D	
z_data = np.array(hrts['Z']['data']) #1D
r_data = np.array(hrts['TE']['x']) #1D
	
for j in range(nb_points):
    thomson.channel[j].position.r=r_data[j]
    thomson.channel[j].position.z=z_data[j] 
    thomson.channel[j].t_e.data=te_data[:,j]
    thomson.channel[j].t_e.data_error_upper=dte_data[:,j]
    thomson.channel[j].n_e.data=ne_data[:,j]
    thomson.channel[j].n_e.data_error_upper=dne_data[:,j]

data_entry.put(thomson)

Reading data

Once the data is saved to ids we can open it and for example plot it:

...

Page tree

Versions Compared

Old Version 15

New Version Current

Key

Saving data to IMAS

IMAS UDA

Saving data to IMAS

Saving Thomson scattering data

Saving Thomson scattering data

Reading data

Page tree

Page History

Versions Compared

Old Version 15

New Version Current

Key

Saving data to IMAS

IMAS UDA

Saving data to IMAS

Saving Thomson scattering data

Saving Thomson scattering data

Reading data