1. Building client application
If you haven't built CLI client yet, jump to section Using Catalogue Client at Gateway - fast track
2. Adding variables into database
Catalog QT ver. 2 has the notion of variable - variables define what can be stored inside database. Putting it simple, variable defines what element of summary IDS should be read from input data and stored inside Catalogue QT. From user's perspective it's just a matter of adding variables into Catalogue QT. This process is quite simple and all you have to do is to run CLI based client with proper description of variables.
2.1. Generating list of variables
The easiest way to generate variables inside Catalogue QT is to use XML transformation and pass the result to CLI based client.
> cd catalog_qt_2/client/catalog-ws-client > xsltproc ../../sql/feed/imas/dumpSummaryFields.xsl $IMAS_PREFIX/include/IDSDef.xml > variables_summary > java -jar ./target/catalogAPI.jar -addVariables --url http://catalog:8080 --variables-file `pwd`/variables_summary > xsltproc ../../sql/feed/imas/dumpFairFields.xsl $IMAS_PREFIX/include/IDSDef.xml > variables_fair > java -jar ./target/catalogAPI.jar -addVariables --url http://catalog:8080 --variables-file `pwd`/variables_fair
Once this action is finished Catalog QT will be aware of all the variables that are supposed to be collected.
2.1.1. Anatomy of variable description
In case you are interested in the format of variable definition, here it goes.
3. Adding request into database
Catalog QT revolves around the concept of so called Request. Due to the fact that import may take time, users don't perform it in a blocking way. Instead users just tell Catalog QT what they want to have stored inside Catalog QT database. Then, requests are processed in background by Catalog QT Update Process.
Submitting new request is quite simple. All you have to do is to run CLI client with -addRequest argument.
Please note that pulse file you want to populate must be visible on the machine where Catalog QT Update Process is running.
> java -jar ./target/catalogAPI.jar -addRequest \ # command to be executed
--url http://catalog:8080 \ # location of Catalog QT WS server (can be on remote site)
--user g2michal \ # name of the user requesting data population
--experiment-uri "mdsplus:/?shot=10;run=1;machine=west;user=g2michal;version=3" # location of input data (currently only local MDSPlus files are supported)
> java -jar ./target/catalogAPI.jar -addRequest \ # command to be executed
--url http://catalog:8080 \ # location of Catalog QT WS server (can be on remote site)
--user g2michal \ # name of the user requesting data population
--experiment-uri "mdsplus:/?shot=10;run=1;machine=west;user=g2michal;version=3;occurence=1" # it is possible to specify particular occurrence of the IDS
3.1. Anatomy of experiment-uri
Why URI in a first place?
We have decided to use URI format due to the fact that Catalog QT will, most probably, support multiple input sources (e.g. UDA, files, HDC based serialised data). Sticking with typical description of source: {shot, run, user, machine, version} was not enough. This is why we have decided to switch to URI based description of data source.
At the moment only locally stored, MDSPlus based, pulse files are supported by Catalog QT. There is an ongoing development to provide support for UDA based source of the data.
4. Listing requests inside Catalog QT
It is possible to list all the requests that were ever stored inside Catalog QT DB. It is possible using command: -listRequests
> java -jar ./target/catalogAPI.jar -listRequests --pretty-format --url http://catalog:8080 ID: [ 1] - Submission date: 26-06-2020 18:32:40 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=1106;run=2020;machine=f4f;user=g2michal;version=3 ID: [ 2] - Submission date: 26-06-2020 18:33:29 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=55102;run=3;version=3;machine=west;user=g2fpi;occurrence=1 ID: [ 3] - Submission date: 26-06-2020 18:33:43 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=55101;run=3;version=3;machine=west;user=g2fpi;occurrence=1 ID: [ 4] - Submission date: 26-06-2020 18:33:51 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=55100;run=3;version=3;machine=west;user=g2fpi;occurrence=1 ID: [ 5] - Submission date: 26-06-2020 18:33:56 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=3106;run=2020;version=3;machine=f4f;user=g2bpalak ID: [ 6] - Submission date: 26-06-2020 18:34:01 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=2106;run=2020;version=3;machine=f4f;user=g2bpalak ID: [ 7] - Submission date: 26-06-2020 18:34:06 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=1206;run=2020;version=3;machine=f4f;user=g2bpalak ID: [ 8] - Submission date: 26-06-2020 18:34:11 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=1106;run=2020;version=3;machine=f4f;user=g2bpalak ID: [ 9] - Submission date: 30-06-2020 13:00:10 - Current status: PROCESSED - experiment URI: mdsplus:/?shot=1106;run=2021;machine=f4f;user=g2michal;version=3
5. Using Catalogue Client at Gateway - fast track
At Gateway there is a dedicated machine that hosts Catalogue QT Web Services - http://catalog:8080
The easiest way to get Catalogue QT client is to run it at Gateway. Here are the steps to run it
# You need IMAS environment with Open JDK 11
> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk
# You have to clone source repository (you need to request access if you haven't done so)
# https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git - and "Request to join project"
> git https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git
Cloning into 'catalog_qt_2'... Cloning into 'catalog_qt_2'...
Username for 'https://gitlab.eufus.psnc.pl': g2afilip
Password for 'https://g2afilip@gitlab.eufus.psnc.pl':
> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file \
-Dfile=${IMAS_PREFIX}/jar/imas.jar \
-DgroupId=imas -DartifactId=imas \
-Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
-DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests
# You are ready to Cataogue QT client application
> java -jar ./target/catalogAPI.jar -help
usage: java -jar ${FULL_PATH_TO_CATALOG_API_JAR}/catalogAPI.jar
-help show help
-listRequests List all available requests
requires: --url
optional: --pretty-format
-addRequest Add new request into database
requires: --url, --experiment-uri,
--user
-listVariables List all the variables defined
inside Catalog
requires: --url
-addVariables Add variables from the file into
Catalog
requires: --url, --variables-file
-startUpdateProcess Start application in Update
Process mode
requires: --url
optional: --wait-time,
--slice-limit
-verbose Be verbose
--url <URL> Catalog API URL
--variables-file <FILE> File with the list of variables
--experiment-uri <EXPERIMENT URI> Input data URI
format:
mdsplus:/?shot=##;run=##;machine=#
#;user=##;version=##
--user <USER_NAME> Name of the user performing the
action
--wait-time <SECONDS> You can specify number of seconds
before Update Process starts to
process next request
--slice-limit <NUMBER_OF_SLICES> You can specify maximum number of
time slices to be populated.
Useful in case of huge input
files.
--pretty-format If applicable, this option will
make the code to print output as
pretty as possible