Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • MySQL installation,
  • Spring Boot installation (execution of Spring Boot based application),
  • Exposing RESTful Web Services.

Schema of components used by RESTful WS server

Starting

...

WS API server from source codes

Catalogue QT Web Services are hosted using Spring-Boot framework. In order to start server you have to clone the repository with source code. Source code of all the components can be downloaded from the code repository (Catalog QT2).

Code Block
> git clone https://YOUR_USER_NAME_AT_GATEWAY@gforge-nextgitlab.eufus.psnc.eupl/gitsimulation_catalogue/catalog_qt_2.git


Info

If you don't have access yet, make sure to request for the access.

...

Code Block
> mysql -u itm_catalog_rw -p -h localhost itm_catalog_qt
Enter password:
...
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6318
Server version: 8.0.20 MySQL Community Server - GPL

...
...

mysql> show tables;
+----------------------------+
| Tables_in_itm_catalog_testqt |
+----------------------------+
| annotation                 |
| catalog_parameters         |
| entry                      |
| entry_data               |
| entry_replacement        |
| experiment                 |
| outcome filter                   |
| referenceoutcome                  |
| replacereference                |
| request    |
| request             |
| request_status           |
| tag                        |
| tag_entry                  |
| user                     |
| user_filter              |
| variable                 |
| variable_type            |
+----------------------------+
1317 rows in set (0.00 sec)


mysql>


Info

In case you have issues while accessing your MySQL installation, please consult your sys admin to resolve the issue.

...

 Starting Spring Web Services

To prepare Catalog QT environment run compile.sh script - it will create server application and basic components. 

...

Info

Note that you can use different port number for `Spring Boot` application

Code Block
> SERVER_PORT=8081 mvn spring-boot:run

 Debugging

In our application we have several different debugging paths depending on the docker environment and what you want to debug.

You can debug our app locally on your local machine using  e g. Intellij IDE.

Our app includes 4 components:

  • Catalog QT Web Services 
  • Update Process
  • Imas Inotify
  • MySQL

We have two docker containers:

  • docker compose - the app is built of independent dockers that are connected to each other. According to the idea of microservices.
  • single-container - mailny used by developers, all components are built on the basis of one Dockerfile, which creates one container.

Let's get through each debbuging path. 

 Local docker-compose debugging

Warning

Things you must do BEFORE building the image:

Catalog WS in docker-compose debugging

To debug catalog-ws-server you need to add specific lines to docker-compose.override.yaml in server section

Code Block
languagejava
 server:
    volumes:
      - ./imasdb:/home/imas/public/imasdb
      - ./build/catalog_qt_2:/catalog_qt_2 #1
    ports:
      - 5005:5005 #2
    environment: 
      - DEBUG=1 #3
 
  1. Maps your code on host machine to the code inside container, allowing you to use your favourite IDE debugger capabilities. Also, you can change your code and run docker-compose restart to rerun container. This allows container to integrate your newest code.

  2. Exposes port for Java debugger, usually 5005.

  3. Enables debugging on catalog-ws-server.

You also need to run

Code Block
echo '' > build/files/server/application.properties.patch

and in catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties change

Code Block
languagejava
themeEclipse
-spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC
+spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

After that you can build the image with debugging WS turned on.

Update Process in docker-compose debugging ???

Imas Inotify in docker-compose debugging ???

Mysql in docker-compose debbuging ???

Local single-container debugging

If you are a Catalog WS developer the easiest and most comfortable way to debug code is use single container as a base image.

You can debug code in two ways:

  • in IDE - you can see only the stack trace of WebServices, which is relevant in most cases
  • inside container - it helps you find out what's happening inside docker while all of the components are working.

IDE debugging

To debug code in IDE  you should prepare your enviroment in such way:

your catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties  should look like this:

Code Block
languagejava
themeEclipse
firstline1
linenumberstrue
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https
server.port=8081  # We have to change ports on our local instance, because 8080 is taken by container
server.http.port=8082
server.http.interface=0.0.0.0

keycloak.realm = ******** 
keycloak.auth-server-url= *******
keycloak.resource= *********
keycloak.realm-key= ********
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# These need to be commented
# If you plan to use HTTPS, make sure to uncomment this one
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

swagger-ui.authorization.header=true 
Warning

Change both ports in lines, because 8080 and 8081 is already taken by running docker container.

Warning

Lines 18 - 21 should be filled with proper data!

...


 Catalogue QT Docker - Docker Compose Installation


This container is desined to simplify installation of Catalogue QT and it's components.
Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.


Known limitations

Note that this container should be used only for research purposes.


Info

Our codes and detailed documentation are open-source and you can find them in here: https://gitlab.eufus.psnc.pl/simulation_catalogue/catalogue_qt_docker


Starting Catalogue Update Process

Catalogue Update Process serves the purpose of reading data (MDSPlus pulse files, UDA, etc.) and populating Catalogue QT with the content of summary IDS. Catalogue Update Process is the only component that requires access to input data and to IMAS infrastructure. This is related to the fact that data are read from the structures created and maintained by IMAS based components.

Requirements

Catalogue QT Update Process requires following components to be installed on target machine:

Code Block
Java - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well

Catalogue QT Update Process

Installation of Catalog QT CLI based client

I assume that you have already cloned the source code repository. In that case, all you have to do follows:

Warning
titleJava based bridge to IMAS

Note that Catalogue QT Update Process requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.


Code Block
> cd catalog_qt_2/client/catalog-ws-client

Make sure to import imas.jar into your local Maven repository. All you have to do is to run following command

Code Block
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${LOCATION_OF_YOUR_IMAS_JAR}/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo

Once imas.jar is in place you can build and run Catalog QT CLI based client.

Code Block
> mvn install -DskipTests
> java -jar ./target/catalogAPI.jar -help

You should see the help screen with available options

Info
titleCatalog QT WS location

For the purpose of this documentation I assume that Catalog QT WS server is installed on localhost:8080.

Starting Catalogue QT Update

Warning

Please note that Update Process requires IMAS environment to work properly. If you don't have your IMAS environment set, it will complain

Code Block
> java -jar ./target/catalogAPI.jar  -startUpdateProcess --url http://localhost:8080
IMAS library not set up in the environment. (IMAS_PREFIX missing)

The easiest way to use IMAS environment is to run Update Process on the machine where IMAS environment is set. For example - Gateway.

Once you have everything in place you can start update process

Code Block
> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

Starting Catalogue Update Process at Gateway - fast track

Info
titleDedicated WS machine

At Gateway there is a dedicated machine that hosts Catalogue QT Web Services - http://catalog:8080


The easiest way to get Catalogue QT Update Process running is to run it at Gateway. Here are the steps to run it

Code Block
# You need IMAS environment with Open JDK 11

> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk

# You have to clone source repository (you need to request access if you haven't done so)
# https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git- and "Request to join project"

> git clone https://gitlab.eufus.psnc.pl/simulation_catalogue/catalog_qt_2.git
Cloning into 'catalog_qt_2'... 
Username for 'https://gitlab.eufus.psnc.pl': g2afilip
Password for 'https://g2afilip@gitlab.eufus.psnc.pl': 

> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${IMAS_PREFIX}/jar/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests

# You are ready to run Update Process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

...

Code Block
docker run -i \
  -p 8080:8080 \ 
  -p 3306:3306 \
  -v `pwd`/imasdb:/home/imas/public/imasdb \
  --name catalogqt_debug_ide -t catalogqt

And in your IDE connect properly to DB and run application in debug mode.

Running tests

You can run unit tests by changing directory to: ws/catalog-ws and running.

Code Block
> mvn test
Info

Test code doesn't use MySQL server based database. It's safe to run tests even after database is already created. Tests will not touch your production database.

Starting Catalogue Update Process

Catalogue Update Process serves the purpose of reading data (MDSPlus pulse files, UDA, etc.) and populating Catalogue QT with the content of summary IDS. Catalogue Update Process is the only component that requires access to input data and to IMAS infrastructure. This is related to the fact that data are read from the structures created and maintained by IMAS based components.

Requirements

Catalogue QT Update Process requires following components to be installed on target machine:

Code Block
Java - openjdk 11.0.2 - you can use JDK from Oracle as well
Apache Maven - 3.6.3 - you can use Apache Maven distributed with project as well

Catalogue QT Update Process

Installation of Catalog QT CLI based client

I assume that you have already cloned the source code repository. In that case, all you have to do follows:

Warning
titleJava based bridge to IMAS

Note that Catalogue QT Update Process requires IMAS based components. You have to import IMAS based JAR file into your local Maven repository. It's not something that will make you suffer. All you have to do is to run one command, and you have to make sure you have imas.jar file.

Code Block
> cd catalog_qt_2/client/catalog-ws-client

Make sure to import imas.jar into your local Maven repository. All you have to do is to run following command

Code Block
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${LOCATION_OF_YOUR_IMAS_JAR}/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo

Once imas.jar is in place you can build and run Catalog QT CLI based client.

Code Block
> mvn install -DskipTests
> java -jar ./target/catalogAPI.jar -help

You should see the help screen with available options

Info
titleCatalog QT WS location

For the purpose of this documentation I assume that Catalog QT WS server is installed on localhost:8080.

Starting Catalogue QT Update

Warning

Please note that Update Process requires IMAS environment to work properly. If you don't have your IMAS environment set, it will complain

Code Block
> java -jar ./target/catalogAPI.jar  -startUpdateProcess --url http://localhost:8080
IMAS library not set up in the environment. (IMAS_PREFIX missing)

The easiest way to use IMAS environment is to run Update Process on the machine where IMAS environment is set. For example - Gateway.

Once you have everything in place you can start update process

Code Block
> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://localhost:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

Starting Catalogue Update Process at Gateway - fast track

Info
titleDedicated WS machine

At Gateway there is a dedicated machine that hosts Catalogue QT Web Services - http://catalog:8080

The easiest way to get Catalogue QT Update Process running is to run it at Gateway. Here are the steps to run it

Code Block
# You need IMAS environment with Open JDK 11

> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk

# You have to clone source repository (you need to request access if you haven't done so)
# https://gforge6.eufus.eu/gf/project/catalog_qt_2/ - and "Request to join project"

> git clone https://gforge6.eufus.eu/git/catalog_qt_2
Cloning into 'catalog_qt_2'...
Username for 'https://gforge6.eufus.eu': YOUR_GW_USER_NAME
Password for 'https://g2michal@gforge6.eufus.eu': YOUR_GW_PASSWORD
> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${IMAS_PREFIX}/jar/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests

# You are ready to run Update Process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

Anatomy of application.properties file

Spring based Web Services are run inside Tomcat server. Configuration of services can be done using application.properties file.

Code Block
# location of database - typically, it will point at localhost, but
# it's also possible to change location of MySQL server.
# Docker based installation (docker-compose) will change it to db:3306
# In case of docker-compose based installation, MySQL is visible as another host
# Note that you don't have to change anything
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

# Default user name and password for database connection. Note that this connection
# will not work (by default) for external hosts. This is why we don't quite care about
# user/pass - however, you can alter these and make sure they don't contain default values
spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

# In case of errors we want to embed error message as well (so we better know what went wrong)
server.error.include-message=always

# We definitely don't want to log SQL queries. However, if you want to see them, feel free
# to enable this 
spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https
server.port=8080
server.http.port=8081
server.http.interface=0.0.0.0

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# If you plan to use HTTPS, make sure to uncomment this one
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

swagger-ui.authorization.header=true