Versions Compared

Old Version 47

changes.mady.by.user Agata Filipczak

Saved on

compared with

New Version Current

changes.mady.by.user Agata Filipczak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • MySQL installation,
  • Spring Boot installation (execution of Spring Boot based application),
  • Exposing RESTful Web Services.

Schema of components used by RESTful WS server

Starting

...

WS API server from source codes

Catalogue QT Web Services are hosted using Spring-Boot framework. In order to start server you have to clone the repository with source code. Source code of all the components can be downloaded from the code repository (Catalog QT2).

Code Block
> git clone https://YOUR_USER_NAME_AT_GATEWAY@gforge-nextgitlab.eufus.psnc.eupl/gitsimulation_catalogue/catalog_qt_2.git


Info

If you don't have access yet, make sure to request for the access.

...

Info

In case you have issues while accessing your MySQL installation, please consult your sys admin to resolve the issue.

...

 Starting Spring Web Services

To prepare Catalog QT environment run compile.sh script - it will create server application and basic components. 

...

Info

Note that you can use different port number for `Spring Boot` application

Code Block
> SERVER_PORT=8081 mvn spring-boot:run

 Startnig WS API Server from Docker Containers

...


 Catalogue QT Docker - Docker Compose Installation


This container is desined to simplify installation of Catalogue QT and it's components.
 Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.


Known limitations

Note that this container should be used only for research purposes.


Info

Our codes and detailed documentation are open-source and you can find them in here:

...

https://

...

We have two docker containers that containes all of the above components.

  • docker compose - the app is built of independent dockers that are connected to each other. According to the idea of microservices.
  • single-container - mainly used by developers, all components are built on the basis of one Dockerfile, which creates one container.

 Docker-compose installation

This container is desined to simplify installation of Catalogue QT and it's components. Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.

...

If you are installing our docker for the first time please go to the next section of this documentation. Otherwise you have access to all of our repositories and can build docker easily as follows:

Code Block
> git clone https://github.com/mkopsnc/catalogue_qt_docker.git
> cd docker-compose/build
> ./build.sh
> cd ..
> ./run.sh -s notoken

Prepare your work environment

In order to build this container, you will need access to few repositories. This container is based on:

...

imas/ual

...

catalog_qt_2

...

dashboard-ReactJS

...

Make sure you can access imas/ual

This Catalogue Qt 2 Docker image is based on imas/ual Docker image. It is available from Docker registry rhus-71.man.poznan.pl.

Before you proceed, make sure you can access the registry. You can test it by executing following command.

Code Block
> docker login rhus-71.man.poznan.pl

You will be asked for a user name and password. If you don't have it, contact developer of this project.

Make sure you can access catalog_qt_2

You will also need an access to `catalog_qt_2` project. Make sure you can access it.

Code Block
> git clone --single-branch develop https://YOUR_USER_NAME@gforge6.eufus.eu/git/catalog_qt_2

You will be asked for a user name and password. If you don't have it, contact developer of this project.

Make sure you can access dashboard-ReactJS

Docker image that contains Dashboard application can be downloaded from a Docker registry registry.apps.man.poznan.pl. Before you proceed, make sure you can access the registry. You can test it by executing following command:

Code Block
> docker login registry.apps.man.poznan.pl/f4f/dashboard-ui/assets:branch-develop
Note

Note! Running Dashboard locally requires an entry inside /etc/hosts

Code Block
127.0.0.1       localhost.dashboard-ui.pl

Make sure you can access imas-watchdog project

This repository is publicly available. All you have to do, is to double check whether you can clone it in docker-compose/build folder.

Code Block
> git clone --single-branch master https://github.com/tzok/imas-watchdog.git

Anatomy of application.properties file

Spring based Web Services are run inside Tomcat server. Configuration of services can be done using application.properties file.

Code Block
# location of database - typically, it will point at localhost, but
# it's also possible to change location of MySQL server.
# Docker based installation (docker-compose) will change it to db:3306
# In case of docker-compose based installation, MySQL is visible as another host
# Note that you don't have to change anything
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

# Default user name and password for database connection. Note that this connection
# will not work (by default) for external hosts. This is why we don't quite care about
# user/pass - however, you can alter these and make sure they don't contain default values
spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

# In case of errors we want to embed error message as well (so we better know what went wrong)
server.error.include-message=always

# We definitely don't want to log SQL queries. However, if you want to see them, feel free
# to enable this property
spring.jpa.show-sql=false

# We don't want to generate DB schema from bean classes
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https

# This is tricky :)
# If server.ssl fields are set, this field defines https port
# If server.ssl fields are not set, this field defines http port
server.port=8080

# However, we need http port anyway (for some components). This is why we expose services on
# http anyway. At the and we can end up with two different configurations
# http  (8080) and http (8081) - no certificates
# https (8443) and http (8081) - certificates 
server.http.port=8081
server.http.interface=0.0.0.0

# ------- Keycloak settings -------
keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# ------- HTTPS settings --------

# If you plan to use HTTPS, make sure to uncomment this one
# You have to make sure to generate and configure SSL certificate for your domain
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

# ------- Bearer token authorization --------

# Should we check authorisation header or not. This feature toggle enables sort of "single user mode"
# It's useful in case you don't have Keycloak and don't care about user roles. Once set to "false"
# it will make Web Services behave as if there is only one user
swagger-ui.authorization.header=true

We have few alternative path to run our app:

docker compose:

Warning

in every docker-compose path the first line should be changed to

spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

-localhost:3306

+db:3306

  1. docker-compose with keycloak authentication and HTTPS settings both enabled, swagger-ui.authorization.header=true
  2. docker-compose with keycloak authentication enabled and HTTPS settings disabled, swagger-ui.authorization.header=true
  3. docker-compose with keycloak authentication disabled and HTTPS settings enabled, swagger-ui.authorization.header=false
  4. docker-compose with keycloak authentication and generated HTTPS settings both disabled, swagger-ui.authorization.header=false

Example of application.properties file for docker-compose: 

Info

Look at the structure:

  • KeyCloak settings are enabled
  • Bearer token authorization is enabled
  • HTTPS settings are disabled
Code Block
# The host is 'db' not localhost!
spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always
spring.mvc.dispatch-options-request=true

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
server.port=8080
server.http.port=8081
server.http.interface=0.0.0.0

# ------ Keycloak settings -------
# If you plan to use Keycloak authorization, make sure to uncomment this one 

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username


# ------ Bearer token authorization settings -------
# If you plan to send autorization token, make sure to make this one 'true'

swagger-ui.authorization.header=true 

# ------ HTTPS settings -------
# If you plan to use HTTPS, make sure to uncomment this one

#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

the same for single container:

  1. single-container with keycloak authentication and HTTPS settings both enabled, swagger-ui.authorization.header=true
  2. single-container with keycloak authentication enabled and HTTPS settings disabled, swagger-ui.authorization.header=true
  3. single-container with keycloak authentication disabled and HTTPS settings enabled, swagger-ui.authorization.header=false,
  4. single-container  with keycloak authentication and generated HTTPS settings both disabled, swagger-ui.authorization.header=false

Example of application.properties file for single-container:

Info

Look at the structure:

  • KeyCloak settings are disabled
  • Bearer token authorization is disabled
  • HTTPS settings are enabled
Code Block
# The host is localhost
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always
spring.mvc.dispatch-options-request=true

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
server.port=8080
server.http.port=8081
server.http.interface=0.0.0.0

# ------ Keycloak settings -------
# If you plan to use Keycloak authorization, make sure to uncomment this one 

#keycloak.realm = fair4fusion-docker-demo
#keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
#keycloak.resource= catalogqt-cli
#keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
#keycloak.bearer-only = true
#keycloak.public-client=true
#keycloak.principal-attribute=preferred_username


# ------ Bearer token authorization settings -------
# If you plan to send autorization token, make sure to make this one 'true'

#swagger-ui.authorization.header=true 

# ------ HTTPS settings -------
# If you plan to use HTTPS, make sure to uncomment this one

server.ssl.key-store=file:///home/imas/cert/keystore.p12
server.ssl.key-store-password=catalogqt
server.ssl.keyStoreType=PKCS12
server.ssl.keyAlias=tomcat

Building and running

The checklist you should make before building the app:

...

Can I succesfully login to rhus-71.man.poznan.pl?

...

Do I have catalog_qt_2 codes?
Did I changed application properties? If yes in your /catalog_qt_2 run script
Code Block
> ./compile.sh

After this you can finally build and run the app!

Docker compose

In order to build and run docker-compose container you have to do following:

Code Block
> cd docker-compose/build
> ./build.sh
> cd ..
> docker-compose up

...

Code Block
> docker-compose run

localhost:8080/swagger-ui.html to access Web Services via Swagger based UI.

Single-container

Once Catalog QT 2 is in place, you can build the single-container.

Code Block
> docker build -t catalogqt .

...

Please note that for tagged release you have to specify tag of the imas-notify project. You can do it following way

Code Block
> docker build -t catalogqt --build-arg INOTIFY_TAG=0.4 .

Starting the container is quite simple, all you have to do is to run

Code Block
> docker run -i -t --name catalogqt_test catalogqt

...

Exposing Spring Boot based Web Services to the outside world

If you want to access Catalog QT WS API outside of the container, you can expose its ports:

Code Block
> docker run -i -t -p 8080:8080 --name catalogqt_test catalogqt

localhost:8080/swagger-ui.html to access Web Services via Swagger based UI.

Setting up SSL certificate

The best way to obtain SSL certificate is to use certbot. You can get certbot in multiple ways described here.

After installation, you need to obtain raw .pem certificate and convert it to .p12. Do this by running 

Code Block
certbot certonly --standalone 

Provide required information about your domain.

Required files will be located in /etc/letsencrypt/live/name_of_your_domain .

Go to this folder and run the command below. 

Code Block
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name tomcat -CAfile chain.pem -caname root

You will be asked to provide a password. Remember it as you will have to enter it in application.properties. The output file keystore.p12 is the file that has all the required information to set up SSL.

In application.properties enter this information:

Code Block
server.ssl.key-store="path to your keystore.p12 file"
server.ssl.key-store-password="password to keystore.p12 file"

Congratulations! You have set up an SSL certificate!

Debugging

In our application we have several different debugging paths depending on the docker environment and what you want to debug.

You can debug our app locally on your local machine using  e g. Intellij IDE.

Let's get through each debbuging path. 

 Local docker-compose debugging

Warning

Things you must do BEFORE building the image:

Catalog WS in docker-compose debugging

To debug catalog-ws-server you need to have  docker-compose.override.yaml file that looks like this (just copy and paste it in your empty file) 

Code Block
languagejava
version: "3.6"

services:
  server:
    volumes:
      - ./imasdb:/home/imas/public/imasdb
      - ./cert:/home/imas/cert
      - ./build/catalog_qt_2:/catalog_qt_2  #1
    ports:
      - 5005:5005  #2
    environment:
      - DEBUG=1   #3

  updateprocess:
    volumes:
      - ./imasdb:/home/imas/public/imasdb

  inotify:
    volumes:
      - ./imasdb:/home/imas/public/imasdb
Info

#1  Maps your code on host machine to the code inside container, allowing you to use your favourite IDE debugger capabilities. Also, you can change your code and run docker-compose restart to rerun container. This allows container to integrate your newest code

#2  Exposes port for Java debugger, usually 5005.

#3  Enables debugging on catalog-ws-server.

You also need to clear patch file:

Code Block
true '' > build/files/server/application.properties.patch

and in catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties change the first line to second line

Code Block
languagejava
themeEclipse
-spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC
+spring.datasource.url=jdbc:mysql://db:3306/itm_catalog_qt?serverTimezone=UTC

After that you can build the image with debugging WS turned on.

Local single-container debugging

If you are a Catalog WS developer the easiest and most comfortable way to debug code is use single container as a base image.

You can debug code in two ways:

  • in IDE - you can see only the stack trace of WebServices, which is relevant in most cases
  • inside container - it helps you find out what's happening inside docker while all of the components are working.

IDE debugging

To debug code in IDE  you should prepare your enviroment in such way:

your catalog_qt_2/server/catalog-ws-server/src/main/resources/application.properties  should look like this:

Code Block
languagejava
themeEclipse
firstline1
linenumberstrue
spring.datasource.url=jdbc:mysql://localhost:3306/itm_catalog_qt?serverTimezone=UTC

spring.datasource.username=itm_catalog_rw
spring.datasource.password=itm_catalog_rw
spring.jpa.properties.hibernate.jdbc.time_zone=UTC

server.error.include-message=always

spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none

# This is additional http handler, on another port
# We need this one, in case we plan to use https
server.port=8081  # We have to change ports on our local instance, because 8080 is taken by container
server.http.port=8082  # This one as well
server.http.interface=0.0.0.0

keycloak.realm = fair4fusion-docker-demo
keycloak.auth-server-url=https://sso.apps.paas-dev.psnc.pl/
keycloak.resource= catalogqt-cli
keycloak.realm-key= MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjOCDGJsBi7rxVjf0RQb8pm0LAGsEKFcH7g7mKSqpFvp1uOypUeiYe5dwlwkXAXaYeYs0J70LB8E6mtVUcykbmp+XrqD1nn3yfPxlVLSg7iCvJqMUq8udsUbsyT3M/32/kssXurgY7rX5JhdtkYeAgq+9ifIjLQZhALg+FvEsX9C+D30WQDAChEljlReb+Y4UTz2aIqz9C+90bqG1ZIX4o3Dli1PZDosTNM444CwDTbrFrenctOTDtGPodo9k2jze8McZFAIrdUYi9mKD8v0frs8NUUW/TQj9h62swXdvVAfzYTd+R7aMRG0eXMV3rJc38DfsCsF7bkqSg0b4l8GcaQIDAQAB
keycloak.bearer-only = true
keycloak.public-client=true
keycloak.principal-attribute=preferred_username

spring.mvc.dispatch-options-request=true

# These need to be commented
# If you plan to use HTTPS, make sure to uncomment this one
#server.ssl.key-store=file:///home/imas/cert/keystore.p12
#server.ssl.key-store-password=catalogqt
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=tomcat

swagger-ui.authorization.header=true
Warning

Change both ports in lines 14 and 15 because ports 8080 and 8081 are already taken by running docker container.

and then run this command:

Code Block
docker run -i \
  -p 8080:8080 \ 
  -p 3306:3306 \
  -v `pwd`/imasdb:/home/imas/public/imasdb \
  --name catalogqt_debug_ide -t catalogqt

And in your IDE connect properly to DB and run application in debug mode.

Running tests

You can run unit tests by changing directory to: ws/catalog-ws and running.

Code Block
> mvn test
Info

Test code doesn't use MySQL server based database. It's safe to run tests even after database is already created. Tests will not touch your production database.gitlab.eufus.psnc.pl/simulation_catalogue/catalogue_qt_docker


Starting Catalogue Update Process

...

Code Block
# You need IMAS environment with Open JDK 11

> module purge
> module load cineca
> module load imasenv
> module unload itm-java
> module load openjdk

# You have to clone source repository (you need to request access if you haven't done so)
# https://gforge6gitlab.eufus.psnc.eupl/gfsimulation_catalogue/project/catalog_qt_2/ .git- and "Request to join project"

> git clone https://gforge6gitlab.eufus.eu/gitpsnc.pl/simulation_catalogue/catalog_qt_2.git
Cloning into 'catalog_qt_2'... 
Username for 'https://gforge6gitlab.eufus.eupsnc.pl': YOUR_GW_USER_NAMEg2afilip
Password for 'https://g2michal@gforge6g2afilip@gitlab.eufus.eupsnc.pl': YOUR_GW_PASSWORD

> cd catalog_qt_2/client/catalog-ws-client/
> mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file  \
    -Dfile=${IMAS_PREFIX}/jar/imas.jar \
    -DgroupId=imas -DartifactId=imas \
    -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar \
    -DlocalRepositoryPath=`pwd`/local-maven-repo
> mvn install -DskipTests

# You are ready to run Update Process

> java -jar ./target/catalogAPI.jar -startUpdateProcess --url http://catalog:8080
[main] INFO pl.psnc.catalog.client.cli.commands.StartUpdateProcess - Getting list of requests for processing from Catalog.
...
...

...