Versions Compared

Old Version 14

changes.mady.by.user Bartosz Palak

Saved on

compared with

New Version 15

changes.mady.by.user Bartosz Palak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The goals of Access Layer

The Access Layer (or AL) if is the central data access library which allows reading, which  allow data access for the users/applications through various APIs and programming languages. 

Thus, its main purpose is to provide mechanisms for reading, writing and manipulating manipulating IDS data data objects, as being defined in the Data Dictionary (DD), through various APIs and programming language. 

—Implemented to allow data access for the users/applications

—AL operates only at the IDS level

—AL allows “atomic” operations such as:

—put or get data (IDS),

—access to single time slices of data (IDS)

  • API providing access methods (read/write) to an ITER physics Database based on the ITER Physics Data Model
  • Provided in Fortran, C++, Matlab, Java, Python
  • The only effort for using the Data Model is to map the input/output of your code to the Data Model and add some GET/PUT commands
  • The access methods are writing to a local database stored in your account
  • These local databases can be shared among users (for reading only) and can be accessed remotely

Access Layer architecture (Bartek)

Image Removed

.

Access Layer architecture (Bartek)

In order to cope with multiple In order to cope with multiple languages and maintaining at the same time a unique structure definition, the AL architecture defines a few layers.

Image Added

Application Layer

Application Layer is the layer of users programs or dedicated tools that manipulates IDS data through High Level Interfaces

High Level Interfaces

 This two layers. The top layer provides the external Application Programming Interface (API), and its code is automatically produced from the XML description of the ITM database structure. For each supported programming language, a high level layer is generated in the target language. The following sections will describe the language specific API, and they provide all the required information for simulation program developers.

High Level Interfaces available in AL include:

  • Fortran
  • C++
  • Matlab
  • Java
  • Python

Methods exposed by High Level Interfaces:

  • —Operations on data base entry
    • —CREATE
    • —OPEN
    • —DELETE
    • —CLOSE
  • —Operations on IDSes  - AL operates at the IDS level (with some exceptions) providing only methods for “atomic” operations such as:
    • —PUT
    • —GET
    • —PUT_SLICE
    • —GET_SLICE

Low Level

The Low Level layer is implemented in CPP (but with C API)  The lower layer is implemented in C and provides unstructured data access to the underlying databaseunderlying databases/backends. It defines an API which is used by all the high level layer implementations. Knowledge of this API (presented in a later section) is not necessary to end users, and is only required to the developers of new language specific high level implementations of the AL as well as the developers of support tools for ITM management.

Application Layer

High Level Interfaces

Low Level

...

Backends

Backends are plug-ins that allows for interaction between an abstract Low Level layer and physical storages.

Currently implemented backends allows to store data in: memory cache, as MDSPlus files, HDF5 files and ASCII files (this BE is used mainly for testing purposes)

High Level Interfaces and their API (Application Programming Interface)

...

Info
titleitmdbs command

Usage: imasdbs [OPTIONS] [COMMAND]

This program lists existing databases.

Possible commands are:

        list <shot number>- list existing databases

       slices <shot number> <run number> - list existing databases, including number of timeslices and time range for time-dependent IDSes

        times <shot number> <run number> - list existing databases, including number of timeslices their time points for time-dependent IDSes

        tokamak - list existing tokamaks (with data versions)                                                                                

        dataversion - list existing dataversions (with tokamaks)                                                                             

If the optional arguments shot number and run number are given, only databases with these numbers will be shown.

If no command is given, the list command is performed.

To see databases stored in the public database, use 'public' as the user name.

Options:

  -h, --help            show this help message and exit

  -u USER, --user=USER  Show databases of specified user

  -t TOKAMAK, --tokamak=TOKAMAK                         

                        Show only databases for specified tokamaks

  -v VERSION, --version=VERSION                                   

                        Show only databases for specified data version

  --backend=BACKEND     Show databases written with given backend(s).  Comma-

                        separated list of backends (Currently supported:     

                        mdsplus, hdf5). By default all backends are shown.   

  -c, --compact         Compact/reduced output



shell> imasdbs -t test slices 9999 2
Tokamak: test
   Data version: 3
      UAL Backend: mdsplus
         Shot    10
             Run:     40
                 core_profiles:   25 slices (345.0 - 345.48)
                  core_sources:   25 slices (345.0 - 345.48)
                core_transport:   25 slices (345.0 - 345.48)
                   equilibrium:   25 slices (345.0 - 345.48)
               transport_solver_numerics:   25 slices (345.0 - 345.48)
                          wall:   25 slices (345.0 - 345.48)


shell> imasdbs  -u palakb
Tokamak: test
   Data version: 3
      UAL Backend: mdsplus
         Shot     1 Runs:     1
         Shot     2 Runs:     3   666   777   999
         Shot    10 Runs:    30    40    42    60    61    64    65    66    80    81    99   123   666   999  1234
         Shot    12 Runs:     1     2    99
         Shot    13 Runs:     1


 Dumping pulse files

To list the content (all data) of an IDS,  use idsdump  script

shell> idsdump
Usage: idsdump <USER> <TOKAMAK> <VERSION> <SHOT> <RUN> <IDS>


shell> idsdump $USER test 3 9999 2 equilibrium
class equilibrium
Attribute ids_properties
        class ids_properties
        Attribute comment:
        Attribute homogeneous_time: 1
        Attribute source:
        Attribute provider:
        Attribute creation_date:
[.......]
Attribute code
        class code
        Attribute name: 12 34 56 78 90
        Attribute commit: 12 34 56 78 90
        Attribute version: 12 34 56 78 90
        Attribute repository: 12 34 56 78 90
        Attribute parameters: 12 34 56 78 90
        Attribute output_flag
        [-819925519  678927020  358961885  263985221 -518535735 -656888240
          885898039 -949201251  187087431  189678740  306846126  536940120
         -842545485 -121858537 -867824798  103609281 -986039164 -761981263
         -444948662 -178414734   91809633  -65221224  575637439 -526052305]
Attribute time
[ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
 19. 20. 21. 22. 23. 24.]

Dumping an IDS node

Getting a subset of an IDS enables reading only a node (and its descendants if the node is a structure), making the GET operation much faster. To retrieve only requested node one should call the script idsdumppath . 

idsdumppath
Usage: idsdumppath <USER> <TOKAMAK> <VERSION> <SHOT> <RUN> <IDS> <DATA_PATH>

Path syntax:

  • The path to requested node(s) is separated by slashes (“/path/to/node(s)”).  
  • Nodes representing arrays must contain indexes (“/path/to/array(idx)/field”) or “Fortran style” indices (“path/to/array(x:y)/field”) 
  • Limitation: In case of nested arrays, it is not allowed to specify set of indices for AoS ancestors. Only given values of AoS ancestors indices are handled: (e.g. “field/with/ancestorAoS(x:y)/field/AoS(n :m)” is not managed)

...

  • “flux_loop(1)/flux/data(1:5)” 
  • “bpol_probe(2:3)/field/data” 
  • “loop(:)/current” 
  • “time(4:-1)”
  • “profiles_1d(2)/grid/rho_tor_norm(2:4)” 



shell> idsdumppath  $USER test 3 9999 2 equilibrium "code"
Type: <class 'imas_3_24_0_ual_4_2_0.equilibrium.code__structure'>
----------------------------------------------
----------------------------------------------
class code
Attribute name: 12 34 56 78 90
Attribute commit: 12 34 56 78 90
Attribute version: 12 34 56 78 90
Attribute repository: 12 34 56 78 90
Attribute parameters: 12 34 56 78 90
Attribute output_flag
[-819925519  678927020  358961885  263985221 -518535735 -656888240
  885898039 -949201251  187087431  189678740  306846126  536940120
 -842545485 -121858537 -867824798  103609281 -986039164 -761981263
 -444948662 -178414734   91809633  -65221224  575637439 -526052305]


shell> idsdumppath $USER test 3 9999 2 equilibrium "code/output_flag(0)"
Type: <class 'numpy.int32'>
----------------------------------------------
----------------------------------------------
-819925519


Copying database files directly

In case you know user name, machine name, shot number and run number, you can import users' database files copying them directly from the users' public directories. Database files are located inside:

~$USERNAME/public/imasdb/$TOKAMAKNAME/$DATAVERSION/0/ids_SSSSRRRR.*


Take a look at the example below. We will copy data from user michalo, machine test, shot: 12 and run: 2


# change directory in your $HOME
cd $HOME/public/imasdb/test/3/0/
 
# copy data files (pay attention to *_dot_* at the end of command line!)
cp ~michalo/public/imasdb/test/3/0/ids_120002.* .
cp ~michalo/public/imasdb/test/3/0/ids_130003.* .


Adapting user code into IMAS - 22.09

...