The purpose of this documentation is to make users aware of how to download, compile and run server side components of Catalogue QT.

1. Table of contents

2. What are we going to do?

This section of documentation explains how to install server part components of Catalogue QT. I am discussing here:

MySQL installation,
Spring Boot installation (execution of Spring Boot based application),
Exposing RESTful Web Services.

Schema of components used by RESTful WS server

3. Starting RESTful based Web Services - Catalogue QT server

Catalogue QT Web Services are hosted using Spring-Boot framework. In order to start server you have to clone the repository with source code. Source code of all the components can be downloaded from the code repository (Catalog QT2).

> git clone https://YOUR_USER_NAME_AT_GATEWAY@gforge-next.eufus.eu/git/catalog_qt_2

If you don't have access yet, make sure to request for the access.

3.1. Requirements

Catalogue QT ver. 2 requires following components to be installed on the machine where it will be started

Java         - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3          - you can use Apache Maven distributed with project as well
mysql        - 8.0.20         - you can use MariaDB instead (we haven't tested it thoug)

3.2. Creating database

Catalog QT ver. 2 requires access to database that will store all the imported data. You have to create this database. Schema for creating database is located inside project structure: sql/schema/catalog_qt_db.sql.

To create DB schema inside mysql you will need root password for your MySQL installation. Then, you can run following:

# WARNING! This line will drop schema: itm_catalog_qt and create it from the scratch. Make sure you are not destroying things!

> mysql -u root -p < sql/schema/a_001_catalog_qt_db.sql
> mysql -u root -p < sql/schema/d_001_update_variables.sql

This script will create two users: itm_catalog_ro/itm_catalog_ro and user itm_catalog_rw/itm_catalog_rw with read-only and read-write privileges, respectively.

It's a good idea to check whether you can access your newly created database. Make sure you can successfully connect there.

> mysql -u itm_catalog_rw -p -h localhost itm_catalog_qt
Enter password:
...
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6318
Server version: 8.0.20 MySQL Community Server - GPL

...
...

mysql> show tables;
+--------------------------+
| Tables_in_itm_catalog_qt |
+--------------------------+
| annotation               |
| catalog_parameters       |
| entry                    |
| entry_data               |
| entry_replacement        |
| experiment               |
| filter                   |
| outcome                  |
| reference                |
| request                  |
| request_status           |
| tag                      |
| tag_entry                |
| user                     |
| user_filter              |
| variable                 |
| variable_type            |
+--------------------------+
17 rows in set (0.00 sec)


mysql>

In case you have issues while accessing your MySQL installation, please consult your sys admin to resolve the issue.

3.3. Starting WS API server

3.3.1. 3.3.1 Starting WS from source codes

To prepare Catalog QT environment run compile.sh script - it will create server application and basic components.

Java based bridge to IMAS

Note that Catalogue QT Server requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.

In order to run Catalogue QT ver. 2 Web Services API server you have to start Spring-Boot application. Make sure to go inside directory: server/catalog-ws-server and execute > java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar

> cd server/catalog-ws-server/target/
> java -jar target/catalog-ws-server-1.1.0-SNAPSHOT.jar
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------< pl.psnc.catalog:catalog-ws >---------------------
[INFO] Building catalog-ws 1.1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
...
...
 .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.0.RELEASE)

2020-06-05 13:21:31.533  INFO 17919 --- [           main] p.p.c.catalogws.CatalogWsApplication     : Starting CatalogWsApplication on catalog.eufus.eu with PID 17919
...
...

Note that you can use different port number for `Spring Boot` application

> SERVER_PORT=8081 mvn spring-boot:run

3.3.2. Startnig WS from docker containers

Our codes are open-source and you can find them in here: https://github.com/mkopsnc/catalogue_qt_docker

Please follow the instruction with cloning just the catalogue_qt_docker.

Make sure that you clones the repo and you have access to catalog_qt_2' codes in /build folder and you can succesfully login to rhus-71.man.poznan.pl.

Dont build anything yet!

Our app includes 4 components:

Catalog QT Web Services
Update Process
Imas Inotify
MySQL

We have two docker containers that containes all of the above components.

docker compose - the app is built of independent dockers that are connected to each other. According to the idea of microservices.
single-container - mailny used by developers, all components are built on the basis of one Dockerfile, which creates one container.

Moreover we have few options with authetincation you can enable/disable before building your own app by uncommenting/commenting particular lines in one file which is:

/docker-compose/build/catalog_qt_2/server/catalog-ws-server/src/main/resources/appliation.properties

3.3.3. Anatomy of application.properties file

Spring based Web Services are run inside Tomcat server. Configuration of services can be done using application.properties file.

# location of database - typically, it will point at localhost, but
# it's also possible to change location of MySQL server.
# Docker based installation (docker-compose) will change it to db:3306
# In case of docker-compose based installation, MySQL is visible as another host
# Note that you don't have to change anything
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

# Default user name and password for database connection. Note that this connection
# will not work (by default) for external hosts. This is why we don't quite care about
# user/pass - however, you can alter these and make sure they don't contain default values
spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

# In case of errors we want to embed error message as well (so we better know what went wrong)
server.error.include-message=always

# We definitely don't want to log SQL queries. However, if you want to see them, feel free
# to enable this property
spring.jpa.show-sql=false

# We don't want to generate DB schema from bean classes
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https

# This is tricky :)
# If server.ssl fields are set, this field defines https port
# If server.ssl fields are not set, this field defines http port
server.port=8080

# However, we need http port anyway (for some components). This is why we expose services on
# http anyway. At the and we can end up with two different configurations
# http  (8080) and http (8081) - no certificates
# https (8443) and http (8081) - certificates 
server.http.port=8081
server.http.interface=0.0.0.0

# ------- Keycloak settings -------
keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# ------- HTTPS settings --------

# If you plan to use HTTPS, make sure to uncomment this one
# You have to make sure to generate and configure SSL certificate for your domain
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

# ------- Bearer token authorization --------

# Should we check authorisation header or not. This feature toggle enables sort of "single user mode"
# It's useful in case you don't have Keycloak and don't care about user roles. Once set to "false"
# it will make Web Services behave as if there is only one user
swagger-ui.authorization.header=true

We have few alternative path to run our app:

docker compose:

in every docker-compose path the first line should be changed to

spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

-localhost:3306

+db:3306

docker-compose with keycloak authentication and HTTPS settings both enabled, swagger-ui.authorization.header=true
docker-compose with keycloak authentication enabled and HTTPS settings disabled, swagger-ui.authorization.header=true
docker-compose with keycloak authentication disabled and HTTPS settings enabled, swagger-ui.authorization.header=false
docker-compose with keycloak authentication and generated HTTPS settings both disabled, swagger-ui.authorization.header=false

Example of application.properties file for docker-compose:

# The host is 'db' not localhost!
spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always
spring.mvc.dispatch-options-request=true

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
server.port=8080
server.http.port=8081
server.http.interface=0.0.0.0

# ------ Keycloak settings -------
# If you plan to use Keycloak authorization, make sure to uncomment this one 

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username


# ------ Bearer token authorization settings -------
# If you plan to send autorization token, make sure to make this one 'true'

swagger-ui.authorization.header=true 

# ------ HTTPS settings -------
# If you plan to use HTTPS, make sure to uncomment this one

#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

the same for single container:

single-container with keycloak authentication and HTTPS settings both enabled, swagger-ui.authorization.header=true
single-container with keycloak authentication enabled and HTTPS settings disabled, swagger-ui.authorization.header=true
single-container with keycloak authentication disabled and HTTPS settings enabled, swagger-ui.authorization.header=false,
single-container with keycloak authentication and generated HTTPS settings both disabled, swagger-ui.authorization.header=false

Example of application.properties file for single-container:

# The host is localhost
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always
spring.mvc.dispatch-options-request=true

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
server.port=8080
server.http.port=8081
server.http.interface=0.0.0.0

# ------ Keycloak settings -------
# If you plan to use Keycloak authorization, make sure to uncomment this one 

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username


# ------ Bearer token authorization settings -------
# If you plan to send autorization token, make sure to make this one 'true'

swagger-ui.authorization.header=true 

# ------ HTTPS settings -------
# If you plan to use HTTPS, make sure to uncomment this one

#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

3.4. Debugging

In our application we have several different debugging paths depending on the docker environment and what you want to debug.

You can debug our app locally on your local machine using e g. Intellij IDE.

Let's get through each debbuging path.

3.4.1. Local docker-compose debugging

Things you must do BEFORE building the image:

3.4.1.1. Catalog WS in docker-compose debugging

To debug catalog-ws-server you need to have docker-compose.override.yaml file that looks like this (just copy and paste it in your empty file)

version: "3.6"

services:
  server:
    volumes:
      - ./imasdb:/home/imas/public/imasdb
      - ./cert:/home/imas/cert
      - ./build/catalog_qt_2:/catalog_qt_2  #1
    ports:
      - 5005:5005  #2
    environment:
      - DEBUG=1   #3

  updateprocess:
    volumes:
      - ./imasdb:/home/imas/public/imasdb

  inotify:
    volumes:
      - ./imasdb:/home/imas/public/imasdb

#1 Maps your code on host machine to the code inside container, allowing you to use your favourite IDE debugger capabilities. Also, you can change your code and run docker-compose restart to rerun container. This allows container to integrate your newest code

#2 Exposes port for Java debugger, usually 5005.

#3 Enables debugging on catalog-ws-server.

You also need to run to clear patch file

true '' > build/files/server/application.properties.patch

and in catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties change

-spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC
+spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

After that you can build the image with debugging WS turned on.

3.4.2. Local single-container debugging

If you are a Catalog WS developer the easiest and most comfortable way to debug code is use single container as a base image.

You can debug code in two ways:

in IDE - you can see only the stack trace of WebServices, which is relevant in most cases
inside container - it helps you find out what's happening inside docker while all of the components are working.

3.4.2.1. IDE debugging

To debug code in IDE you should prepare your enviroment in such way:

your catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties should look like this:

spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https
server.port=8081  # We have to change ports on our local instance, because 8080 is taken by container
server.http.port=8082
server.http.interface=0.0.0.0

keycloak.realm = ******** 
keycloak.auth-server-url= *******
keycloak.resource= *********
keycloak.realm-key= ********
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# These need to be commented
# If you plan to use HTTPS, make sure to uncomment this one
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

swagger-ui.authorization.header=true

Change both ports in lines 14 and 15 because ports 8080 and 8081 are already taken by running docker container.

Lines from 18 to 21 should be filled with proper data!

and then run this command:

docker run -i \
  -p 8080:8080 \ 
  -p 3306:3306 \
  -v `pwd`/imasdb:/home/imas/public/imasdb \
  --name catalogqt_debug_ide -t catalogqt

And in your IDE connect properly to DB and run application in debug mode.

3.5. Running tests

You can run unit tests by changing directory to: ws/catalog-ws and running.

> mvn test

Test code doesn't use MySQL server based database. It's safe to run tests even after database is already created. Tests will not touch your production database.

4. Starting Catalogue Update Process

Catalogue Update Process serves the purpose of reading data (MDSPlus pulse files, UDA, etc.) and populating Catalogue QT with the content of summary IDS. Catalogue Update Process is the only component that requires access to input data and to IMAS infrastructure. This is related to the fact that data are read from the structures created and maintained by IMAS based components.

4.1. Requirements

Catalogue QT Update Process requires following components to be installed on target machine:

Java - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well

4.2. Catalogue QT Update Process

4.2.1. Installation of Catalog QT CLI based client

I assume that you have already cloned the source code repository. In that case, all you have to do follows:

Java based bridge to IMAS

Note that Catalogue QT Update Process requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.

> cd catalog_qt_2/client/catalog-ws-client

Make sure to import imas.jar into your local Maven repository. All you have to do is to run following command

> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${LOCATION_OF_YOUR_IMAS_JAR}/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo

Once imas.jar is in place you can build and run Catalog QT CLI based client.

> mvn install -DskipTests
> java -jar ./target/catalogAPI.jar -help

You should see the help screen with available options

Catalog QT WS location

For the purpose of this documentation I assume that Catalog QT WS server is installed on localhost:8080.

4.2.2. Starting Catalogue QT Update

Please note that Update Process requires IMAS environment to work properly. If you don't have your IMAS environment set, it will complain

> java -jar ./target/catalogAPI.jar  -startUpdateProcess --url http://localhost:8080
IMAS library not set up in the environment. (IMAS_PREFIX missing)

The easiest way to use IMAS environment is to run Update Process on the machine where IMAS environment is set. For example - Gateway.

Once you have everything in place you can start update process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

5. Starting Catalogue Update Process at Gateway - fast track

Dedicated WS machine

At Gateway there is a dedicated machine that hosts Catalogue QT Web Services - http://catalog:8080

The easiest way to get Catalogue QT Update Process running is to run it at Gateway. Here are the steps to run it

# You need IMAS environment with Open JDK 11

> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk

# You have to clone source repository (you need to request access if you haven't done so)
# https://gforge6.eufus.eu/gf/project/catalog_qt_2/ - and "Request to join project"

> git clone https://gforge6.eufus.eu/git/catalog_qt_2
Cloning into 'catalog_qt_2'...
Username for 'https://gforge6.eufus.eu': YOUR_GW_USER_NAME
Password for 'https://g2michal@gforge6.eufus.eu': YOUR_GW_PASSWORD
> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${IMAS_PREFIX}/jar/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests

# You are ready to run Update Process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

Page tree

Administration