...
Code Block | ||||
---|---|---|---|---|
| ||||
In order to access the new mdsplus.jetdata.eu service, users will need to make a minor change to their codes, changing the host parameter in the connection calls from "mdsplus.jet.efda.org" or "mdsplus.jet.uk" to "ssh://<username>@mdsplus.jetdata.eu" where <username> is the your JET account username (e.g. jsmith or xy1234) Users will also need to register their SSH public key with us in order for this to work. Please do this by emailing support@it.ukaea.uk with the subject line “MDSplus SSH Key Registration”. Please put your SSH Public key into the body of the message – do not add attachments to the email. It is important that you provide your public key only and do not include your private key (users should take all reasonable steps to protect their private keys). (Due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used) Suggested email text to use: Please register my SSH Public key for use with mdsplus.jetdata.eu. My username is : <your shortname> My SSH Public key is: Once your key has been registered we will confirm this back to you and you will then be able to test the connection and your clients If you need information about creating SSH keys please see the information here: • There is some general information here: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-2 • There is a page on how to create key pairs with openssh at https://www.ssh.com/academy/ssh/keygen • There is a page on how to create key pairs with putty at https://www.ssh.com/academy/ssh/putty/windows/puttygen • The www.ssh.com has pages about generating keys for other ssh clients too. Important Note: when generating SSH key pairs, we’d suggest that most users will find it more convenient not to use a passphrase (i.e. leave it blank). Also, due to a limitation of the mdsplus server, please avoid using Ed25519 keys, though ecdsa keys can be used |
Once the pair of keys 'id_rsa_jet' is created it necessary that the system can distinguish between the keys. You can inform the system by editing 'config' file in ~/ .ssh folder.
Code Block | ||
---|---|---|
| ||
Host jet
HostName mdsplus.jetdata.eu
User <your jet username>
IdentityFile ~/.ssh/id_rsa_jet |
In an example we fetch JET data (dda='hrts', uid = 'jetppf', seq=0, dtype= ['TE', 'DTE', 'NE', 'DNE', 'Z']) .
Example of script fetching HRTS JET data.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
#example of saving experimental data to ids import os import json def read_ppf(conn, shot, ppf, seq=0, uid='jetppf',debug=False): ierr = conn.get('_sig=ppfuid("' + uid +'")') c = '_sig=jet("ppf/%s/%d",%d)'% (ppf, seq, shot) if debug: print('\nDEBUG: %s\n' % c) try: s = conn.get(c) raw = s.data() dim0 = conn.get('dim_of(_sig,0)').data() try: dim1 = conn.get('dim_of(_sig,1)').data() return {'raw': raw, 'x':dim0, 't':dim1} except: return {'raw': raw, 't':dim0} except: return None def readExperimental(data_exp, host, server='ssh://'): ''' reads experimental data after connecting to the host via ssh server data_exp: data details in dictionary format ''' import sys try: import MDSplus haveMDS = True except: print(' No MDSplus support found.\n') print('\n\n\n Exiting...\n') exit(1) try: conn = MDSplus.Connection(server+host) print('Connection OK') connected = True except: print('in readExperimental, MDSplus failed: ',sys.exc_info()[1]) connected = False if connected: try: EXP={} for sig in data_exp['dtype']: print('reading dtype: ',sig) signal= data_exp['dda']+'/'+sig aux = read_ppf(conn, discharge, signal, seq=data_exp['seq'], uid=data_exp['uid']) #conv# = read_ppf(conn,args.Shot, 'WSXP/RHOT', seq=args.seq, uid=args.uid) #print("rhot:dx ", conv['x']-aux['x']) #print('rhot[raw]:', convZ coordinate is one dimensional, aux['raw']) print('len(x): ',len(aux['x'])) print('time: ',aux['t'][0],aux['t'][-1],len(aux['t'])) print('shape(value):',aux['raw'].shape)is 2D by default print('len(aux)', len(aux))if sig=='Z': print('') aux['raw'] #EXP=None= aux['raw'].flatten() EXP[sig] = { 'time': aux['t'].tolist(), 'x':aux['x'].tolist(), 'data': aux['raw'].tolist(), 'signal': signal #'rhot' : np.sqrt(conv['raw']) } except: print('in readExperimental, read_ppf failed: ',sys.exc_info()[1]) return None del(conn) else: EXP=None return EXP ########### host='jet' discharge=99357 # data to download data={} data['dda']='hrts' data['uid'] = 'jetppf' data['seq']=0 data['dtype']=['TE','DTE','NE','DNE','Z'] EXP=readExperimental(data,host) if FalseEXP: a a_file = open("'data_jet_ssh_fetched.json"_fetched_'+str(discharge)+'_'+data['dda']+'_'+str(data['seq'])+'.json', "w") json.dump(EXP, a_file) a_file.close() a_file) a_file.close() |
IMAS UDA
It is possible to access and fetch and map data from several experiments WEST, JET , TCV and AUG using UDA protocol which id described in detailed on ITER confulence pages (requires iter account). Accessing data with UDA needs to be adapted due to recent (Aug 2021) change in connection protocol.
Saving data to IMAS
Before saving the experimental data to imas it is important check which version of imas data dictonary we are using.
...
Code Block | ||
---|---|---|
| ||
import json import imas,os,datetime,sys import getpass import numpy as np from imas import imasdef db = 'data_access_tutorial' shot=99357 run=1 #creates the Data Entry object 'data_entry', a kind of handler of the pulse file with sho, run, belonging to database 'data_access_tutorial' of the current user, using the MDS+ backend data_entry = imas.DBEntry(imasdef.MDSPLUS_BACKEND, db, shot, run, user_name=getpass.getuser()) #data_entry = imas.DBEntry(imasdef.HDF5_BACKEND, db, shot, run, user_name=getpass.getuser()) # Open save data_entry to data base # Tries to open data_entry op = data_entry.open() #open() and create(0 return a tuple (x_int,y_int) where x<0, y>0, x number of failures, y number of successes in the current session. #if open fails, create data_entry if op[0]<0: cp=data_entry.create() if cp[0]==0: print("data entry created") elif op[0]==0: print("data entry opened") # Open file with fetched data, the data is not numpy array format with open("data/data_jet_hrts_99357.json") as json_file: hrts=json.load(json_file) x_coord=np.array(hrts['TE']['x']) # no. of space points nb_points = len(x_coord) #no of time slices nb_slices=len(hrts['TE']['time']) #creating the 'thomson_scattering' auxiliary IDS and initializing it thomson = imas.thomson_scattering() #creates a 'thomson scattering' IDS thomson.ids_properties.homogeneous_time=1 #setting the homogeneous time (mandatory) thomson.ids_properties.comment='IDS created for testing the IMAS Data Access layer' #setting the ids_properties.comment attribute #thomson.time=np.array([0.]) #the time(vector) basis must be not empty if homogeneous_time==1 otherwise an error will occur at runtime # since all data is available we can save whole time vector at once thomson.time=np.array(hrts['TE']['time']) thomson.ids_properties.creation_date = datetime.datetime.now().strftime("%y-%m-%d") # the number of channel corresponds to number of data points thomson.channel.resize(nb_points) for j in range(nb_points): thomson.channel[j].t_e.data.resize(1) thomson.channel[j].t_e.data_error_upper.resize(1) thomson.channel[j].n_e.data.resize(1) thomson.channel[j].n_e.data_error_upper.resize(1) # python interface accepts only numpy arrays to be saved in ids te_data = np.array(hrts['TE']['data']) #2D dte_data = np.array(hrts['DTE']['data'])#2D ne_data = np.array(hrts['NE']['data'])#2D dne_data = np.array(hrts['DNE']['data'])#2D z_data = np.array(hrts['Z']['data']) #1D r_data = np.array(hrts['TE']['x']) #1D for j in range(nb_points): thomson.channel[j].position.r=r_data[j] thomson.channel[j].position.z=z_data[j] thomson.channel[j].t_e.data=te_data[:,j] thomson.channel[j].t_e.data_error_upper=dte_data[:,j] thomson.channel[j].n_e.data=ne_data[:,j] thomson.channel[j].n_e.data_error_upper=dne_data[:,j] sequence = hrts['TE']['seq'] data_entry.put(thomson,seq)# the last number is the occurence which can be used to store the data sequence number #closing the Data Entry data_entry.close() |
Note, that we used (once and outside the loop) only one command 'data_entry.put(thomson)' to save the data. Since we had all data available at once we didn't need to use put and putSlice commands to save time slice by time slice. However, the two approaches should be equivalent and provide the same ids. The first one is faster as procedure of saving the data in the physical memory is performed only once.
...