1. Table of contents

2. What are we going to do?

This section of documentation explains how to install server part components of Catalogue QT. I am discussing here:

• MySQL installation,
• Spring Boot installation (execution of Spring Boot based application),
• Exposing RESTful Web Services.

3. Starting WS API server from source codes

Catalogue QT Web Services are hosted using Spring-Boot framework. In order to start server you have to clone the repository with source code. Source code of all the components can be downloaded from the code repository (Catalog QT2).

> git clone https://YOUR_USER_NAME_AT_GATEWAY@gforge-next.eufus.eu/git/catalog_qt_2

3.1. Requirements

Catalogue QT ver. 2 requires following components to be installed on the machine where it will be started

Java         - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3          - you can use Apache Maven distributed with project as well
mysql        - 8.0.20         - you can use MariaDB instead (we haven't tested it thoug)

3.2. Creating database

Catalog QT ver. 2 requires access to database that will store all the imported data. You have to create this database. Schema for creating database is located inside project structure: sql/schema/catalog_qt_db.sql.

To create DB schema inside mysql you will need root password for your MySQL installation. Then, you can run following:

# WARNING! This line will drop schema: itm_catalog_qt and create it from the scratch. Make sure you are not destroying things!

> mysql -u root -p < sql/schema/a_001_catalog_qt_db.sql
> mysql -u root -p < sql/schema/d_001_update_variables.sql

This script will create two users: itm_catalog_ro/itm_catalog_ro and user itm_catalog_rw/itm_catalog_rw with read-only and read-write privileges, respectively.

It's a good idea to check whether you can access your newly created database. Make sure you can successfully connect there.

> mysql -u itm_catalog_rw -p -h localhost itm_catalog_qt
Enter password:
...
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6318
Server version: 8.0.20 MySQL Community Server - GPL

...
...

mysql> show tables;
+--------------------------+
| Tables_in_itm_catalog_qt |
+--------------------------+
| annotation               |
| catalog_parameters       |
| entry                    |
| entry_data               |
| entry_replacement        |
| experiment               |
| filter                   |
| outcome                  |
| reference                |
| request                  |
| request_status           |
| tag                      |
| tag_entry                |
| user                     |
| user_filter              |
| variable                 |
| variable_type            |
+--------------------------+
17 rows in set (0.00 sec)


mysql>

3.3.  Starting Spring Web Services

To prepare Catalog QT environment run compile.sh script - it will create server application and basic components. 

In order to run Catalogue QT ver. 2 Web Services API server you have to start Spring-Boot application. Make sure to go inside directory: server/catalog-ws-server and execute > java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar

> cd server/catalog-ws-server/target/
> java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------< pl.psnc.catalog:catalog-ws >---------------------
[INFO] Building catalog-ws 1.1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
...
...
 .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.0.RELEASE)

2020-06-05 13:21:31.533  INFO 17919 --- [           main] p.p.c.catalogws.CatalogWsApplication     : Starting CatalogWsApplication on catalog.eufus.eu with PID 17919
...
...

> SERVER_PORT=8081 mvn spring-boot:run

4.  Catalogue QT Docker - Docker Compose Installation

This container is desined to simplify installation of Catalogue QT and it's components.
Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.


4.1. Known limitations

Note that this container should be used only for research purposes.

> git clone https://github.com/mkopsnc/catalogue_qt_docker.git
> cd docker-compose/build
> ./build.sh
> cd ..
> ./run.sh -s notoken

4.2. Prepare your work environment

In order to build this container, you will need access to few repositories. This container is based on:

• imas/ual

• catalog_qt_2

• dashboard-ReactJS

• imas-watchdog

4.2.1. Make sure you can access the imas/ual

This Catalogue Qt 2 Docker image is based on the imas/ual Docker image. It is available from Docker registry rhus-71.man.poznan.pl.

Before you proceed, make sure you can access the registry. You can test it by executing the following command.

> docker login rhus-71.man.poznan.pl

You will be asked for a user name and password. If you don't have it, contact the developer of this project.

4.2.2. Make sure you can access catalog_qt_2

You will also need access to the `catalog_qt_2` project. Make sure you can access it.

> git clone --single-branch develop https://YOUR_USER_NAME@gforge6.eufus.eu/git/catalog_qt_2 

You will be asked for a user name and password. If you don't have it, contact the developer of this project.

4.2.3. Make sure you can access dashboard-ReactJS

Docker image that contains Dashboard application can be downloaded from a Docker registry registry.apps.man.poznan.pl. Before you proceed, make sure you can access the registry. You can test it by executing the following command:

> docker login registry.apps.man.poznan.pl/f4f/dashboard-ui/assets:branch-develop

127.0.0.1       localhost.dashboard-ui.pl

4.2.4. Make sure you can access the imas-watchdog project

This repository is publicly available. All you have to do is to double-check whether you can clone it in docker-compose/build folder.

> git clone --single-branch master https://github.com/tzok/imas-watchdog.git

4.3. Building container

In order to build and run the container you have to do the following:

> cd docker-compose/build
> ./build.sh

4.4. Starting container

Catalogue QT 2 Docker can be run using multiple configurations. By default, we provide the following configurations

all-conf    - all avaiable configuration in one file (you can comment particular lines to disable functionalities)
production  - configured for production Keycloak instance (eduTEAMS)
development - configured for development based Keycloak instance (user:pass - demo001:demo001)
debug       - configured for running Docker compose in debug mode (Web Services, Update Process, Scheduler)
notoken     - configured for running Docker compose in single-user mode (no tokens are used for authorization/authentication)

You can run given configuration by calling:

> cd docker-compose
# ./run.sh -s <configuration file suffix> e.g.
> ./run.sh -s notoken

To access our application please paste these URLs in your browser:

• http://localhost:8080/swagger-ui.html to access Web Services via Swagger-based UI.
• http://localhost.dashboard-ui.pl:9100/dashboard/ to access Dashboard-ReactJS

4.5. Configuration

4.5.1. Docker-compose Configuration

Important!!

If you're not familiar with docker enviroment, please remember that building and running are diffrent!
If you have already build container, you can change some of the configuration of containers without rebulding whole docker, which save a lot of time.
To do so open configutation file `docker-compose._deployment_name_.yml` and change if you wish:

• The path where MySQL will store the data (default: $(pwd)/db-data)
• The path where pulsefiles are stored on the host (default: $(pwd)/imasdb)
• To map MySQL port to host port, so you can access the database from the container (by default no ports are exposed)
• To add custom configuration of Web Services: application.properties file

Additionally, you can edit existing configurations, or create your own e.g docker-compose.myconf.yml and run it!

> ./run.sh -s myconf

4.5.2. Catalog QT 2 Web Services Configuration

Moreover, in our catalog-ws-server we have application.properties file, which is a configuration for our Web Services in Springboot. 

These Web Services are run inside the Tomcat server.

4.5.2.1. Anatomy of application.properties file

# location of database - typically, it will point at localhost, but
# it's also possible to change location of MySQL server.
# Docker based installation (docker-compose) will change it to db:3306
# In case of docker-compose based installation, MySQL is visible as another host
# Note that you don't have to change anything
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

# Default user name and password for database connection. Note that this connection
# will not work (by default) for external hosts. This is why we don't quite care about
# user/pass - however, you can alter these and make sure they don't contain default values
spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

# In case of errors we want to embed error message as well (so we better know what went wrong)
server.error.include-message=always

# We definitely don't want to log SQL queries. However, if you want to see them, feel free
# to enable this property
spring.jpa.show-sql=false

# We don't want to generate DB schema from bean classes
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https

# This is tricky :)
# If server.ssl fields are set, this field defines https port
# If server.ssl fields are not set, this field defines http port
server.port=8080

# However, we need http port anyway (for some components). This is why we expose services on
# http anyway. At the and we can end up with two different configurations
# http  (8080) and http (8081) - no certificates
# https (8443) and http (8081) - certificates 
server.http.port=8081
server.http.interface=0.0.0.0

# ------- Keycloak settings -------

keycloak.enabled=true

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# ------- HTTPS settings --------

# If you plan to use HTTPS, make sure to uncomment this one
# You have to make sure to generate and configure SSL certificate for your domain
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

# ------- Bearer token authorization --------

# Should we check authorisation header or not. This feature toggle enables sort of "single user mode"
# It's useful in case you don't have Keycloak and don't care about user roles. Once set to "false"
# it will make Web Services behave as if there is only one user
swagger-ui.authorization.header=true

The default configuration is inside our project, but (before building) if you want to use a different configuration (e.g enabling SSL certificates, enabling Keycloak) you can paste it in folder /catalogue_qt_docker/docker-compose/build/files/server another application.properties file, which will have higher priority and would override existing file in source codes. After this, you can build and run our docker.

If you have already built a container and want to change Web Services configuration, you can do that without rebuilding the docker!
All you need to do is to add application.properties file to this folder docker-compose/volumes/server-properties.

Then add this to your docker-compose._deployment_name_.yml in server section:

server:
  volumes:
    - ./volumes/server-properties:/home/imas/server-properties

This line maps your application.properties on localhost to the file on container.
When the container is taken off, it will have the highest priority.

After changing the settings, it may be necessary to restart from scratch:

> docker-compose rm
> docker-compose up

4.6. Adding persistent storage

You can add persistent storage by setting it up inside docker-compose.yml file

services:
  db:
    volumes:
      - ./volumes/mysql:/var/lib/mysql

It is not required to link ./volumes/mysql location. In case you are using some other location for persistent data, feel free to use it instead.

4.7. Importing data from pulse file

Catalog QT Demonstrator allows importing MDSPlus based data automatically into SQL database. In order to do this, you have to bind mount a volume. In layman terms, you have to tell Docker that you want to make your local filesystem available inside the Docker container. The easiest way to do it is to create a directory (or symbolic link) to an MDSPlus compatible local database.

First of all, make sure you have MDSPlus like directory structure with pulse files. The easiest way to execute Docker container with sample data is to get sample data from box.psnc.pl - these are completely artificially created data prepared by the testing framework.

> curl -s -o f4f_data.tar.gz \
    https://box.psnc.pl/seafhttp/files/01953e73-8ad3-4277-be71-57b69c395355/f4f_data.tar.gz

Make sure your directories structure looks like this:

.
`-- catalogue_qt_docker
    `-- docker-compose
        `-- volumes
            `-- imasdb
                |-- f4f
                |   `-- 3
                |       |-- 0
                |       |   |-- ids_11062020.characteristics
                |       |   |-- ids_11062020.datafile
                |       |   |-- ids_11062020.populate
                |       |   |-- ids_11062020.tree
                |       |   |-- ids_11062021.characteristics
                |       |   |-- ids_11062021.datafile
                |       |   |-- ids_11062021.populate
                |       |   `-- ids_11062021.tree
                |       |-- 1
                |       |-- 2
                |       |-- 3
                |       |-- 4
                |       |-- 5
                |       |-- 6
                |       |-- 7
                |       |-- 8
                |       `-- 9
                `-- script.sh

Directory catalogue_qt_docker/docker-compose/volumes/imasdb is automatically mounted inside the Docker container. It means that anything you will put in will be visible inside the Docker container whenever it is running. Once Docker is running you can schedule data population by creating a file with *.populate extension. You can do it the following way. Inside directory with the data, execute script.sh with the name of the database you want to have populated.

> cd catalogue_qt_docker/docker-compose/volumes/imasdb
> ./script.sh f4f

If anything goes wrong, please delete all the .populate files by executing this command on Linux:

> find . -type f -name "*.populate" -delete

4.8. Setting up an SSL certificate

The best way to obtain an SSL certificate is to use certbot. You can get certbot in multiple ways described here.

After installation, you need to obtain the raw .pem certificate and convert it to .p12. Do this by running 

certbot certonly --standalone 

Provide required information about your domain.

Required files will be located in /etc/letsencrypt/live/name_of_your_domain .

Go to this folder and run the command below. 

openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name tomcat -CAfile chain.pem -caname root

You will be asked to provide a password. Remember it as you will have to enter it in the application.properties. The output file keystore.p12 is the file that has all the required information to set up SSL.

In application.properties enter this information:

server.ssl.key-store="path to your keystore.p12 file"
server.ssl.key-store-password="password to keystore.p12 file"

Congratulations! You have set up an SSL certificate!

4.9. Developer informations

4.9.1. Debugging in docker-compose

You can debug either all the Java-based components, inside the Docker container, or you can specify which one should be started in debug mode. For debugging Java code inside Docker containers we are using JDWP protocol, and by default we are using the following ports

Web Services   - 32889
Update process - 32888
imas-watchdog  - 32887

In order to enable debug mode, you can either use predefined docker-compose.debug.yml or enable debug mode for each service separately by adding sections inside your YAML file of choice.

4.9.2. Catalog-ws-server

To debug catalog-ws-server you need to add the following lines to docker-compose.####.yml in server section

  server:
    volumes:
      - ./volumes/imasdb:/home/imas/public/imasdb
      - /path/to/your/directory/catalog_qt_2/server/catalog-ws-server:/catalog_qt_2/server/catalog-ws-server #1
    ports:
      - 32889:32889 
    environment: 
      - DEBUG_SPRING_BOOT=true

If you want to develop Catalog QT 2 codes in a easy way with connection to container you should:

1. Clone the repo of catalog_qt_2 outside catalogue_qt_docker directory (e.g. /Desktop).
2. Run ./compile.sh script in chosen folder directory.
3. Map your choosen directory path on localhost to directory of catalog_qt_2 codes on docker-compose_server_1 container directory . This is what #1 line is doing.
4. Rerun container.
5. You will see that Spring isn't taking off - that means it waits for a remote debbuger to connect!
6. Go to your IDE - we are using Intellij IDE.
   1. top left corner -> click + Add new configuration
   2. choose Remote JVM Debug
   3. set settings as in a screen shot above (Important!! change port to 32889 or diffrent one set in docker-compose.####.yml
   4.  run debug mode in IDE
7. In your konsole you will see that Spring is taken off!
8. Go to your IDE and set breakpoint
9. In Swagger or Postman send proper request on port 8080
   
   If this works - bravo!! You're ready to debug!

4.9.3. Update process

To debug update-process you need to add the following lines to docker-compose.####.yml in updateprocess section

 updateprocess:
    volumes:
      - ./volumes/imasdb:/home/imas/public/imasdb
    ports:
      - 32888:32888
    environment:
      - DEBUG_UPDATE_PROCESS=true

4.9.4.  Watchdog

To debug imas-watchdog you need to add the following lines to docker-compose.####.yml in updateprocess section

  watchdog:
    volumes:
      - ./volumes/imasdb:/home/imas/public/imasdb
      - ./volumes/fair4fusion-docker-demo:/docker-entrypoint-properties.d
    ports:
      - 32887:32887
    environment:
      - DEBUG_IMAS_WATCHDOG=true

4.10. Running tests

You can run unit tests by changing the directory to: ws/catalog-ws and running.

> mvn test

5. Starting Catalogue Update Process

Catalogue Update Process serves the purpose of reading data (MDSPlus pulse files, UDA, etc.) and populating Catalogue QT with the content of summary IDS. Catalogue Update Process is the only component that requires access to input data and to IMAS infrastructure. This is related to the fact that data are read from the structures created and maintained by IMAS based components.

5.1. Requirements

Catalogue QT Update Process requires following components to be installed on target machine:

Java - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well

5.2. Catalogue QT Update Process

5.2.1. Installation of Catalog QT CLI based client

I assume that you have already cloned the source code repository. In that case, all you have to do follows:

> cd catalog_qt_2/client/catalog-ws-client

Make sure to import imas.jar into your local Maven repository. All you have to do is to run following command

> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${LOCATION_OF_YOUR_IMAS_JAR}/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo

Once imas.jar is in place you can build and run Catalog QT CLI based client.

> mvn install -DskipTests
> java -jar ./target/catalogAPI.jar -help

You should see the help screen with available options

5.2.2. Starting Catalogue QT Update

> java -jar ./target/catalogAPI.jar  -startUpdateProcess --url http://localhost:8080
IMAS library not set up in the environment. (IMAS_PREFIX missing)

Once you have everything in place you can start update process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

6. Starting Catalogue Update Process at Gateway - fast track

The easiest way to get Catalogue QT Update Process running is to run it at Gateway. Here are the steps to run it

# You need IMAS environment with Open JDK 11

> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk

# You have to clone source repository (you need to request access if you haven't done so)
# https://gforge6.eufus.eu/gf/project/catalog_qt_2/ - and "Request to join project"

> git clone https://gforge6.eufus.eu/git/catalog_qt_2
Cloning into 'catalog_qt_2'...
Username for 'https://gforge6.eufus.eu': YOUR_GW_USER_NAME
Password for 'https://g2michal@gforge6.eufus.eu': YOUR_GW_PASSWORD
> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${IMAS_PREFIX}/jar/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests

# You are ready to run Update Process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...