Components
Installation of Catalog QT requires multiple components to be installed and configured to be fully operational. This is due to the fact that numerous, external, components are required during build procedure and during runtime. All these elements are described below.
1. Installation of MySQL/MariaDB
Before you proceed with installation of Catalog QT (CLI/UI) you have to make sure there is a working MySQL server and you have a connection from Web Server and machine where CLI commands will be executed.
Make sure you can connect to server like this:
# -u ual_user_s - you can use different user name # -h $HOST_NAME - this is Gateway based host, but you can have something else here # it depends on your target installation # itm_scheduler_db - database name - you can decide on something else # but, by default, name of catalog db is 'itm_scheduler_db' # # Note that result of the command below might be different for you, depending # on whether you use MySQL or MariDB. Please, make sure that client library # for mysql will have the same version as your server. # # Note that your client may be executed on a completely different machine # than server # > mysql -u ual_user_s -h $HOST_NAME itm_scheduler_db Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 25031 Server version: 5.6.36-log MySQL Community Server (GPL) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
This documentation doesn't specify details on how to install and configure MySQL/MariaDB. There is only a requirement that you need:
one account with root privileges - you will have to create db schema
one account with normal user privileges - this user must have access to all tables inside DB
2. Installation of CLI based components
2.1. Installation of gSOAP 2.8
In order to install CLI components you need to install and configure gSOAP 2.8 library. Please follow instructions here, and make sure gSOAP library is available in your system: gsoap_2.8.70.zip
It's a good idea to provide separate module for gSOAP library that sets some basic locations
module-whatis gsoap <http://http://gsoap2.sourceforge.net/> setenv GSOAP put_the_location_of_gsoap_here # e.g. /some_dir/libs/gsoap/2.8 prepend-path PATH put_location_of_bin_dir_here # e.g. /some_dir/libs/gsoap/2.8/bin prepend-path PKG_CONFIG_PATH put_location_of_pkg_config_here # e.g. /some_dir/libs/gsoap/2.8/lib/pkgconfig
Installation of IMAS UAL
Current version of Catalog QT requires at least IMAS/3.23.2/AL/4.1.5 . Make sure to have IMAS properly installed and configured at your target host. Please note that this manual describes steps for building codes based on branch - R2-newLL it is assumed that codes will use New Low Level features.
2.2. Installation of CLI tools
2.2.1. Checking out repository
Make sure you can access repository: https://gforge6.eufus.eu/svn/catalog_qt, before proceeding any further.
> svn co https://gforge6.eufus.eu/svn/catalog_qt
Catalog QT tools are divided into small sub-projects
. |-- C4gsoap - gSOAP communication layer in C++ |-- catalogScheduler - command line for data submission (called by user) |-- catalogTools - helper classes shared between projects |-- catalogUpdateProcess - process that actually performs data collection (executed on server machine) |-- dbAPI - API for database calls - shared between projects |-- docs - documentation |-- examples - sample files |-- modules - module for the project (used by users) |-- resources - place, where gSOAP layer is generated |-- scripts - helper scripts (store_data.sh, store_machine.sh) `-- src - UI - whole server part (UI); described later in this specification
CatalogQT version
Since now, I am referring to sources inside: https://gforge6.eufus.eu/svn/catalog_qt/branches/R2-newLL
In following section I assume that
$HOST_NAME - points to location of MySQL/MariaDB server
user_with_root_access - is a name of MySQL's user name with root privileges
$UAL_USER - is the name of the user who has access to itm_scheduler_db
(this user is create using create_scheduler_db.sql script)
I am using -p option while calling mysql - it means that you will be asked for password each time you want to log in to MySQL machine. Thus, it is not possible to use following samples in batch mode.
2.2.2. Creating DB schema
Catalog QT tool requires multiple databases to run properly.
- Scheduler DB - this database defines all the machine dependent libraries
- Machine DB - this database contains data from the pulse file for given machine (there might be many machine databases)
2.2.2.1. Creating scheduler DB schema
Before compiling the code, make sure you can create db schema and that you can access it as user. You can create DB schema following way
> cd docs/dbschema > mysql -h $HOST_NAME -u user_with_root_access itm_scheduler_db -p < drop_scheduler_user.sql > mysql -h $HOST_NAME -u user_with_root_access itm_scheduler_db -p < create_scheduler_db.sql # Make sure you can access itm_scheduler_db DB > mysql -h $HOST_NAME -u $UAL_USER itm_scheduler_db -p Enter password: ... ... mysql> show tables; +----------------------------+ | Tables_in_itm_scheduler_db | +----------------------------+ | access_list | | db_authorization | | scheduler | | token_list | +----------------------------+
2.2.2.2. Creating machine dependent DB schema
For each machine (e.g.: pedestal, west, jettc) you have to have separate database schema.
> cd docs/dbschema
# this script will produce SQL script for creating machine
# dependent databases
> ./create_db.sh --db-user=ual_user_001 \
--db-name=jettc \
--host-name=$HOST_NAME \
--schema-name=itm_catalog_001 \
--ws-address=https://address.of.itmcatalog.installation/itmcatalog/api \
--storage=/this/is/the/location/where/all/users/can/write/to
> mysql -h $HOST_NAME -u user_with_root_access -p < drop_db_user_modified.sql
> mysql -h $HOST_NAME -u user_with_root_access -p < create_db_modified.sql
# Once database is created, make sure you can log in
> mysql -h $HOST_NAME -u ual_user_001 -p itm_catalog_001
Enter password:
...
...
mysql> show tables;
+---------------------------+
| Tables_in_itm_catalog_001 |
+---------------------------+
| annotations |
| catalog_parameters |
| entry |
| entry_data |
| intervals |
| machines |
| outcome |
| scheduler |
| users |
| variables |
+---------------------------+
10 rows in set (0.00 sec)
2.2.2.3. Populating database with variables
Each machine DB has to be populated with variables that are supposed to be stored inside DB. At the moment, machine dependent DB has to store all the names of fields stored inside summary IDS. This can be done following way
> xsltproc ./getFieldsIMAS.xsl \
$IMAS_HOME/core/installer/src/$IMAS_VERSION/ual/$UAL_VERSION/xml/IDSDef.xml | \
grep "summary" > \
variables.sql
> mysql -h $HOST_NAME -u ual_user_001 itm_catalog_001 -p < variables.sql
# Make sure you can see variables inside DB
> mysql -h $HOST_NAME -u ual_user_001 itm_catalog_001 -p
Enter password:
...
...
mysql> select count(*) from variables;
+----------+
| count(*) |
+----------+
| 2511 |
+----------+
1 row in set (0.00 sec)
2.2.3. Compilation of CLI tools
gSOAP and WSDL
gSOAP library requires WSDL location used for compilation of the codes. It is not possible to proceed any further without server components. This is related to the fact that gSOAP library requires WSDL location, where SOAP based specification is stored.
Make sure to change directory to branches/R2-newLL.
# Make sure to load IMAS environment - following command may be system dependent
> module load imasenv
> imasdb test
# For bash
> export CATALOG_QT_HOME=`pwd`
> export CATALOG_QT_STORAGE=/this/is/the/location/where/all/users/can/write/to
> export CATALOG_QT_DB_PROPERTIES=${CATALOG_QT_HOME}/dbAPI
# For tcsh
> setenv CATALOG_QT_HOME `pwd`
> setenv CATALOG_QT_STORAGE /this/is/the/location/where/all/users/can/write/to
> setenv CATALOG_QT_DB_PROPERTIES ${CATALOG_QT_HOME}/dbAPI
# setting target machine
#
# unfortunately, there are few places where machine dependent locations are used
# at the moment, two options are available:
#
# setenv MACHINE gateway
# setenv MACHINE jet
#
# it might be, we need yet another one, with new machine settings
# make sure to create and properly configure gSOAP module
# this way, you will be able to use pkg-config, etc.
> module load gsoap/2.8
> make
3. Installation of UI - web client
Make sure you can access repository: https://gforge6.eufus.eu/svn/catalog_qt
> svn co https://gforge6.eufus.eu/svn/catalog_qt
The tool requires the following components to work:
- MySQL database management system
- Apache HTTP Server
- PHP server-side scripting language
3.1. Installation of server tools
The administrator should install the following packages on the server:
- mysql-server or mariadb-server
- php
- libapache2-mod-php
- apache2
3.2. Installation of web client
UI codes are available as a sub-project in the src directory
src |-- api_1_0 - API routines |-- client - nusoap wrappers |-- config.php - main configuration file |-- inc - auxiliary scripts |-- nusoap - nusoap library |-- test.php - nusoap testing script `-- ui2 - main UI scripts and web page
To install the UI, copy the src folder to the apache directory (usually it is /var/www/html/, but it depends on the version and type of operating system)
It is best to add an alias in the httpd.conf file to access the page after a short link without entering the entire address.
After all, restart the apache server:
> service apache2 restart
3.3. Configuration of UI
the fastest way:
Because the codes are mainly prepared for operation on the GW machine, to complete the configuration, all the names of the GW server (http://portal.eufus.eu, https://portal.eufus.eu) should be replaced with the names of the new server.
These names should be located and be replaced in these files:
./src/ui2/config.php ./src/config.php
You can configure access to the mysql database in a ./src/ui2/config.php file (The data should match those given in the section Creating db schema and Installation of MySQL/MariaDB).
Also configuration of API databases access should be similarly adapted:
./src/config_scheduler_db.php
After all, restart the apache server:
> service apache2 restart
3.4. Access to databases by API
The latest version of the API allows communication with many databases through one unified interface.
Thanks to this, there is no need to create a separate API instance for each database on the server side.
The client can query one WS address to have access to all databases available on the server.
In the client's SOAP request header, the variable Qdb contains the database name.
The diagram below shows how this mechanism works:
The API reads the Qdb variable from the header only if the server needs this information. Otherwise, server ignores client's header: