Page History

...

This container is desined to simplify installation of Catalogue QT and it's components.
Instead of installing it on `IMAS` compatible platform you can use it on virtually any machine.


Known limitations

Note that this container should be used only for research purposes.

...

Catalogue QT 2 Docker can be run using multiple configurations. By default, we provide the following configurations

all-conf    - all avaiable configuration in one file (you can comment particular lines to disable functionalities)
production  - configured for production Keycloak instance (eduTEAMS)
development - configured for development based Keycloak instance (user:pass - demo001:demo001)
debug       - configured for running Docker compose in debug mode (Web Services, Update Process, Scheduler)
notoken     - configured for running Docker compose in single-user mode (no tokens are used for authorization/authentication)

...

Docker-compose Configuration

Important!!

If you're not familiar with docker enviroment, please remember that building and running are diffrent!
If you have already build container, you can change some of the configuration of containers without rebulding whole docker, which save a lot of time.
To do so open configutation file `docker-compose._deployment_name_.yml` and change if you wishYou can edit docker-compose._deployment_name_.yml to change:

• The path where MySQL will store the data (default: $(pwd)/db-data)
• The path where pulsefiles are stored on the host (default: $(pwd)/imasdb)
• To map MySQL port to host port, so you can access the database from the container (by default no ports are exposed)
• To add custom configuration of Web Services: application.properties file

...

If you have already built a container and want to change Web Services configuration, you can do that without rebuilding the docker!
All you need to do is to add application.properties file to this folder docker-compose/volumes/server-properties.When the container is taken off, it will have the highest priority.

Then add this to your docker-compose._deployment_name_.yml in server section:

server:
  volumes:
    - ./volumes/server-properties:/home/imas/server-properties

This line maps your application.properties on localhost to the file on container.
When the container is taken off, it will have the highest priority.

After changing the settings, it may be After changing the settings, it may be necessary to restart from scratch:

> docker-compose rm
> docker-compose up

Setting up an SSL certificate

The best way to obtain an SSL certificate is to use certbot. You can get certbot in multiple ways described here.

After installation, you need to obtain the raw .pem certificate and convert it to .p12. Do this by running 

certbot certonly --standalone 

Provide required information about your domain.

Required files will be located in /etc/letsencrypt/live/name_of_your_domain .

Go to this folder and run the command below. 

openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name tomcat -CAfile chain.pem -caname root

You will be asked to provide a password. Remember it as you will have to enter it in the application.properties. The output file keystore.p12 is the file that has all the required information to set up SSL.

In application.properties enter this information:

server.ssl.key-store="path to your keystore.p12 file"
server.ssl.key-store-password="password to keystore.p12 file"

Congratulations! You have set up an SSL certificate!

Adding persistent storage

You can add persistent storage by setting it up inside docker-compose.yml file

services:
  db:
    volumes:
      - ./volumes/mysql:/var/lib/mysql

It is not required to link ./volumes/mysql location. In case you are using some other location for persistent data, feel free to use it instead.

Importing data from pulse file

Catalog QT Demonstrator allows importing MDSPlus based data automatically into SQL database. In order to do this, you have to bind mount a volume. In layman terms, you have to tell Docker that you want to make your local filesystem available inside the Docker container. The easiest way to do it is to create a directory (or symbolic link) to an MDSPlus compatible local database.

First of all, make sure you have MDSPlus like directory structure with pulse files. The easiest way to execute Docker container with sample data is to get sample data from box.psnc.pl - these are completely artificially created data prepared by the testing framework.

> curl -s -o f4f_data.tar.gz \
    https://box.psnc.pl/seafhttp/files/01953e73-8ad3-4277-be71-57b69c395355/f4f_data.tar.gz

Make sure your directories structure looks like this:

Adding persistent storage

You can add persistent storage by setting it up inside docker-compose.yml file

services:
  db:
    volumes:
      - ./volumes/mysql:/var/lib/mysql

It is not required to link ./volumes/mysql location. In case you are using some other location for persistent data, feel free to use it instead.

Importing data from pulse file

Catalog QT Demonstrator allows importing MDSPlus based data automatically into SQL database. In order to do this, you have to bind mount a volume. In layman terms, you have to tell Docker that you want to make your local filesystem available inside the Docker container. The easiest way to do it is to create a directory (or symbolic link) to an MDSPlus compatible local database.

First of all, make sure you have MDSPlus like directory structure with pulse files. The easiest way to execute Docker container with sample data is to get sample data from box.psnc.pl - these are completely artificially created data prepared by the testing framework.

> curl -s -o f4f_data.tar.gz \
    https://box.psnc.pl/seafhttp/files/01953e73-8ad3-4277-be71-57b69c395355/f4f_data.tar.gz

Make sure your directories structure looks like this:

.
`-- catalogue_qt_docker
    `-- docker-compose
        `-- volumes
            `-- imasdb
                |-- f4f
                |   `-- 3
                |       |-- 0
                |       |   |-- ids_11062020.characteristics
     

.
`-- catalogue_qt_docker
    `-- docker-compose
        `-- volumes
  |       |   `|-- imasdb
ids_11062020.datafile
                |       |   |-- f4f
ids_11062020.populate
                |       |   `|-- 3ids_11062020.tree
                |       |   |-- 0 ids_11062021.characteristics
                |       |   |-- ids_1106202011062021.characteristicsdatafile
                |       |   |-- ids_1106202011062021.datafilepopulate
                |       |   |`-- ids_1106202011062021.populatetree
                |       |   |-- ids_11062020.tree1
                |       |   |-- ids_11062021.characteristics2
                |       |   |-- ids_11062021.datafile3
                |       |   |-- ids_11062021.populate4
                |       |   `-- ids_11062021.tree5
                |       |-- 16
                |       |-- 27
                |       |-- 38
                |       |`-- 4
 9
                `-- script.sh

Directory catalogue_qt_docker/docker-compose/volumes/imasdb is automatically mounted inside the Docker container. It means that anything you will put in will be visible inside the Docker container whenever it is running. Once Docker is running you can schedule data population by creating a file with *.populate extension. You can do it the following way. Inside directory with the data, execute script.sh with the name of the database you want to have populated.

>          |       |-- 5
                |       |-- 6
                |       |-- 7
                |       |-- 8
                |       `-- 9
                `-- script.sh

Directory catalogue_qt_docker/docker-compose/volumes/imasdb is automatically mounted inside the Docker container. It means that anything you will put in will be visible inside the Docker container whenever it is running. Once Docker is running you can schedule data population by creating a file with *.populate extension. You can do it the following way. Inside directory with the data, execute script.sh with the name of the database you want to have populated.

> cd catalogue_qt_docker/docker-compose/volumes/imasdb
> ./script.sh f4f

If anything goes wrong, please delete all the .populate files by executing this command on Linux:

> find . -type f -name "*.populate" -delete

Debugging in docker-compose

You can debug either all the Java-based components, inside the Docker container, or you can specify which one should be started in debug mode. For debugging Java code inside Docker containers we are using JDWP protocol, and by default we are using the following ports

Web Services   - 32889
Update process - 32888
imas-watchdog  - 32887

Image Removed

In order to enable debug mode, you can either use predefined docker-compose.debug.yml or enable debug mode for each service separately by adding sections inside your YAML file of choice.

Catalog-ws-server

To debug catalog-ws-server you need to add the following lines to docker-compose.####.yml in server section

...

cd catalogue_qt_docker/docker-compose/volumes/imasdb
> ./script.sh f4f

If anything goes wrong, please delete all the .populate files by executing this command on Linux:

> find . -type f -name "*.populate" -delete

Setting up an SSL certificate

The best way to obtain an SSL certificate is to use certbot. You can get certbot in multiple ways described here.

After installation, you need to obtain the raw .pem certificate and convert it to .p12. Do this by running 

certbot certonly --standalone 

Provide required information about your domain.

Required files will be located in /etc/letsencrypt/live/name_of_your_domain .

Go to this folder and run the command below. 

openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name tomcat -CAfile chain.pem -caname root

You will be asked to provide a password. Remember it as you will have to enter it in the application.properties. The output file keystore.p12 is the file that has all the required information to set up SSL.

In application.properties enter this information:

server.ssl.key-store="path to your keystore.p12 file"
server.ssl.key-store-password="password to keystore.p12 file"

Congratulations! You have set up an SSL certificate!

Developer informations

Debugging in docker-compose

You can debug either all the Java-based components, inside the Docker container, or you can specify which one should be started in debug mode. For debugging Java code inside Docker containers we are using JDWP protocol, and by default we are using the following ports

Web Services   - 32889
Update process - 32888
imas-watchdog  - 32887

Image Added

In order to enable debug mode, you can either use predefined docker-compose.debug.yml or enable debug mode for each service separately by adding sections inside your YAML file of choice.

Catalog-ws-server

To debug catalog-ws-server you need to add the following lines to docker-compose.####.yml in server section

  server:
    volumes:
      - ./volumes/imasdb:/home/imas/public/imasdb
      - /path/to/your/directory/catalog_qt_2/server/catalog-ws-server:/catalog_qt_2/server/catalog-ws-server #1
    ports:
      - 32889:32889 
    environment: 
      - DEBUG_SPRING_BOOT=true

If you want to develop Catalog QT 2 codes in a easy way with connection to container you should:

1. Clone the repo of catalog_qt_2 outside catalogue_qt_docker directory (e.g. /Desktop).
2. Run ./compile.sh script in chosen folder directory.
3. Map your choosen directory path on localhost to directory of catalog_qt_2 codes on docker-compose_server_1 container directory . This is what #1 line is doing.
4. Rerun container.
5. You will see that Spring isn't taking off - that means it waits for a remote debbuger to connect!
6. Go to your IDE - we are using Intellij IDE.Image Added
   1. top left corner -> click + Add new configuration
   2. choose Remote JVM Debug
   3. set settings as in a screen shot above (Important!! change port to 32889 or diffrent one set in docker-compose.####.yml
   4.  run debug mode in IDE
7. In your konsole you will see that Spring is taken off!
8. Go to your IDE and set breakpoint
9. In Swagger or Postman send proper request on port 8080
   
   If this works - bravo!! You're ready to debug!

...

Update process

To debug update-process you need to add the following lines to docker-compose.####.yml in updateprocess section

...