The purpose of this documentation is to make users aware of how to download, compile and run server side components of Catalogue QT.
1. Table of contents
2. What are we going to do?
This section of documentation explains how to install server part components of Catalogue QT. I am discussing here:
- MySQL installation,
- Spring Boot installation (execution of Spring Boot based application),
- Exposing RESTful Web Services.
3. Starting WS API server from source codes
Catalogue QT Web Services are hosted using Spring-Boot framework. In order to start server you have to clone the repository with source code. Source code of all the components can be downloaded from the code repository (Catalog QT2).
> git clone https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git
If you don't have access yet, make sure to request for the access.
3.1. Requirements
Catalogue QT ver. 2 requires following components to be installed on the machine where it will be started
Java - openjdk 11.0.2 - you can use JDK from Oracle as well Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well mysql - 8.0.20 - you can use MariaDB instead (we haven't tested it thoug)
3.2. Creating database
Catalog QT ver. 2 requires access to database that will store all the imported data. You have to create this database. Schema for creating database is located inside project structure: sql/schema/catalog_qt_db.sql.
To create DB schema inside mysql you will need root password for your MySQL installation. Then, you can run following:
# WARNING! This line will drop schema: itm_catalog_qt and create it from the scratch. Make sure you are not destroying things! > mysql -u root -p < sql/schema/a_001_catalog_qt_db.sql > mysql -u root -p < sql/schema/d_001_update_variables.sql
This script will create two users: itm_catalog_ro/itm_catalog_ro and user itm_catalog_rw/itm_catalog_rw with read-only and read-write privileges, respectively.
It's a good idea to check whether you can access your newly created database. Make sure you can successfully connect there.
> mysql -u itm_catalog_rw -p -h localhost itm_catalog_qt Enter password: ... Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 6318 Server version: 8.0.20 MySQL Community Server - GPL ... ... mysql> show tables; +--------------------------+ | Tables_in_itm_catalog_qt | +--------------------------+ | annotation | | catalog_parameters | | entry | | entry_data | | entry_replacement | | experiment | | filter | | outcome | | reference | | request | | request_status | | tag | | tag_entry | | user | | user_filter | | variable | | variable_type | +--------------------------+ 17 rows in set (0.00 sec) mysql>
In case you have issues while accessing your MySQL installation, please consult your sys admin to resolve the issue.
3.3. Starting Spring Web Services
To prepare Catalog QT environment run compile.sh script - it will create server application and basic components.
Java based bridge to IMAS
Note that Catalogue QT Server requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.
In order to run Catalogue QT ver. 2 Web Services API server you have to start Spring-Boot application. Make sure to go inside directory: server/catalog-ws-server and execute > java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar
> cd server/catalog-ws-server/target/ > java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar [INFO] Scanning for projects... [INFO] [INFO] ---------------------< pl.psnc.catalog:catalog-ws >--------------------- [INFO] Building catalog-ws 1.1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- ... ... . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.0.RELEASE) 2020-06-05 13:21:31.533 INFO 17919 --- [ main] p.p.c.catalogws.CatalogWsApplication : Starting CatalogWsApplication on catalog.eufus.eu with PID 17919 ... ...
Note that you can use different port number for `Spring Boot` application
> SERVER_PORT=8081 mvn spring-boot:run
4. Catalogue QT Docker - Docker Compose Installation
This container is desined to simplify installation of Catalogue QT and it's components. Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.
4.1. Known limitations
Note that this container should be used only for research purposes.
Our codes and detailed documentation are open-source and you can find them in here: https://gitlab.eufus.psnc.pl/simulation_catalogue/catalogue_qt_docker
5. Starting Catalogue Update Process
Catalogue Update Process serves the purpose of reading data (MDSPlus pulse files, UDA, etc.) and populating Catalogue QT with the content of summary IDS. Catalogue Update Process is the only component that requires access to input data and to IMAS infrastructure. This is related to the fact that data are read from the structures created and maintained by IMAS based components.
5.1. Requirements
Catalogue QT Update Process requires following components to be installed on target machine:
Java - openjdk 11.0.2 - you can use JDK from Oracle as well Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well
5.2. Catalogue QT Update Process
5.2.1. Installation of Catalog QT CLI based client
I assume that you have already cloned the source code repository. In that case, all you have to do follows:
Java based bridge to IMAS
Note that Catalogue QT Update Process requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.
> cd catalog_qt_2/client/catalog-ws-client
Make sure to import imas.jar into your local Maven repository. All you have to do is to run following command
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file \
-Dfile=${LOCATION_OF_YOUR_IMAS_JAR}/imas.jar \
-DgroupId=imas -DartifactId=imas \
-Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
-DlocalRepositoryPath=`pwd`/local-maven-repo
Once imas.jar is in place you can build and run Catalog QT CLI based client.
> mvn install -DskipTests > java -jar ./target/catalogAPI.jar -help
You should see the help screen with available options
Catalog QT WS location
For the purpose of this documentation I assume that Catalog QT WS server is installed on localhost:8080.
5.2.2. Starting Catalogue QT Update
Please note that Update Process requires IMAS environment to work properly. If you don't have your IMAS environment set, it will complain
> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080 IMAS library not set up in the environment. (IMAS_PREFIX missing)
The easiest way to use IMAS environment is to run Update Process on the machine where IMAS environment is set. For example - Gateway.
Once you have everything in place you can start update process
> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080 [main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog. [main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog. [main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog. ... ...
6. Starting Catalogue Update Process at Gateway - fast track
Dedicated WS machine
At Gateway there is a dedicated machine that hosts Catalogue QT Web Services - http://catalog:8080
The easiest way to get Catalogue QT Update Process running is to run it at Gateway. Here are the steps to run it
# You need IMAS environment with Open JDK 11
> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk
# You have to clone source repository (you need to request access if you haven't done so)
# https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git- and "Request to join project"
> git clone https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git
Cloning into 'catalog_qt_2'...
Username for 'https://gitlab.eufus.psnc.pl': g2afilip
Password for 'https://g2afilip@gitlab.eufus.psnc.pl':
> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file \
-Dfile=${IMAS_PREFIX}/jar/imas.jar \
-DgroupId=imas -DartifactId=imas \
-Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
-DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests
# You are ready to run Update Process
> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...