Preamble

• Repository with Actor Testing Framework: https://gitlab.eufus.psnc.pl/ach/actor_testing_framework

 Framework directory structure

 Your repository:

Assuming your repository looks something like this:

Project_root
├── root_files.extension
├── root_directiories/
│   ├── directories/
│   ├── ...
│   └── files.extension
└── tests # <- this is directory where the whole actor testing framework should be placed

Then the actor testing framework has to be in a directory called tests/ and this is mandatory.

 Actor Testing Framework repository - inside your project:

The inside of the tests/ directory, inside your repository, should contain the entire actor testing framework repository like structure below:

tests
├── __init__.py
├── ci_scripts
│   ├── .gitlab-ci.yml
│   └── .iter-ci.sh
├── configs
│   ├── actor_configuration.yml
│   └── ids_configuration.yml
├── input_pulse_file
│   ├── ITER
│   │   └── 3
│   │       └── 0
│   │           └── {pulsefiles}
│   └── GITLAB
│       └── 3
│           └── 0
│               └── {pulsefiles}
├── main_code_run
│   └── run_actor.sh
├── modules
│   ├── modules_gitlab.yml
│   └── modules_iter.yml
├── native_codes_build
│   └── build_actor.sh
├── pytest_scripts
│   ├── conftest.py
│   ├── __init__.py
│   ├── pytest.ini
│   └── test_actor.py
├── README.md
├── test.sh
└── utils
    ├── get_actor_info.py
    ├── get_ids_info.py
    └── get_ids_name.py

CI engine - ITER or GITLAB

Actor Testing Framework is prepared to run on both ITER CI (called Bamboo) and GITLAB CI (works like Gateway).

There is a slight difference between these platforms that causes the framework to be configured differently to run tests on each of them.

 ITER CI Configuration

 ITER Modules

To set up the ITER modules needed to build and run the actor, there is a file called modules/modules_iter.yml , where you need to provide all the necessary modules, preferably full versions.

├── modules
│   ├── modules_gitlab.yml
│   └── modules_iter.yml	# <-- This is file dedicated for ITER modules

##############################################
- example: inside modules_iter.yml:
##############################################
modules:
  - IMAS
  - iWrap
  - INTERPOS/9.1.0-intel-2020b
  - XMLlib/3.3.1-intel-2020b

 Configuration files

 Actor configuration

There is YAML file in the configs directory called actor_configuration.yml that contain metadata about the actor being tested. For now, only the fields: actor name and relative path from the project root to the actor description file are to be set.

├── configs
│   ├── actor_configuration.yml	# <-- this file
│   └── ids_configuration.yml

##############################################
- example: inside actor_configuration.yml:
##############################################
actor_name: NAME
actor_path: actor/description/file.yaml

 IDS configuration

The Actor Testing Framework assumes that the user needs to prepare input files, output files, and expected (reference) IDS pulse files to be used in the execution of the workflow and tests. On the user side, these files can be placed in the public, user, or custom imasdb path, but only the public and custom imasdb path can be used for the remote CI. This is set in the ids_configuration.yml file and must match the pulse files and IDS used by the actor workflow.

├── configs
│   ├── actor_configuration.yml
│   └── ids_configuration.yml 	# <-- this file

##############################################
- example: inside ids_configuration.yml:
############################################## 
input_ids: 
  user: ./tests/input_pulse_file
output_ids:
  database: ITER
  user: ./tests/input_pulse_file/
  shot: 114100
  run: 42
expected_ids:
  database: ITER
  user: ./tests/input_pulse_file/
  shot: 114100
  run: 43

For your convenience, the Actor Testing Framework comes with a custom imasdb, located under: testing/input_pulse_file/ITER directory path that must be accessed from the relative path of the project root directory.

CI script

 Repository side CI scripts

ITER has a convention to put all CI-related scripts in the ci_scripts/ directory. Actor Testing Framework follows this convention, but can be extended with an additional file in the root directory: ci_scripts/ci_atf.sh. Then, in this upper CI script, you need to execute the lower file: tests/ci_scripts/.iter-ci.sh.

Project_root
├── ci_scripts/
│   ├── ci_atf.sh	# <-- Upper CI script
│   ├── ...
│   └── ci_...sh
│
├── root_files.extension
├── root_directiories/
│
└── tests/
 	└── ci_scripts/
  		└── .iter-ci.sh	# <-- Lower CI script 

##############################################
- example: inside upper ci script: ci_atf.sh
##############################################
#!/bin/sh

./$(dirname $0)/../tests/ci_scripts/.iter-ci.sh

 Remote ITER CI side

On the remote side of ITER CI - Bamboo CI, you need to create a dedicated job for the Actor Testing Framework and a task to run the CI script. The image below shows how such a task can be configured with the top CI script.

Image Modified

 Building Actor Native Codes

Inside the Actor Testing Framework test directory there is a dedicated directory for building native actor code instructions called: native_codes_build/. By default, there is a build_actor.sh script that should provide step-by-step instructions on how to build an actor. For example, it could be a simple make command, but first you need to change the directory your codes are in, basically it should be done the same way you build the codes from the root directory.

tests
├── native_codes_build
│   └── build_actor.sh	# <-- This file
.
.
.

 Run Actor

Inside the Actor Testing Framework test directory there is a dedicated directory to run the actor code, usually it will be a workflow, called main_code_run/. By default, there is a run_actor.sh script that should provide step-by-step instructions for starting an actor or executing a workflow. Basically, it should reflect the same way you run actor from the root directory.

tests 
├── main_code_run
│   └── run_actor.sh	# <-- This file
.
.
.

 Test Cases - Pytest

Actor Testing Framework runs tests using the Python Pytest framework. This requires the user to have a basic understanding of the Pytest framework. All query related scripts are placed in the tests/pytest_scripts directory. The user can provide as many test cases as needed and is responsible for writing them.

tests
├── pytest_scripts
│   ├── conftest.py
│   ├── __init__.py
│   ├── pytest.ini
│   └── test_actor.py
.
.
.

 GILTAB CI Configuration

 GITLAB Modules

To set up the GITLAB modules needed to build and run the actor, there is a file called modules/modules_gitlab.yml , where you need to provide all the necessary modules, preferably full versions.

├── modules
│   ├── modules_gitlab.yml # <-- This is file dedicated for GITLAB modules
│   └── modules_iter.yml

##############################################
- example: inside modules_gitlab.yml:
##############################################
modules:   
  - cineca
  - imasenv/3.35.0/intel/17.0/rc
  - iwrap/0.4.1-py3-10

 Configuration files

 Actor configuration

There is YAML file in the configs directory called actor_configuration.yml that contain metadata about the actor being tested. For now, only the fields: actor name and relative path from the project root to the actor description file are to be set.

├── configs
│   ├── actor_configuration.yml	# <-- this file
│   └── ids_configuration.yml

##############################################
- example: inside actor_configuration.yml:
##############################################
actor_name: NAME
actor_path: actor/description/file.yaml

 IDS configuration

The Actor Testing Framework assumes that the user needs to prepare input files, output files, and expected (reference) IDS pulse files to be used in the execution of the workflow and tests. On the user side, these files can be placed in the public, user, or custom imasdb path, but only the public and custom imasdb path can be used for the remote CI. This is set in the ids_configuration.yml file and must match the pulse files and IDS used by the actor workflow.

├── configs
│   ├── actor_configuration.yml
│   └── ids_configuration.yml 	# <-- this file

##############################################
- example: inside ids_configuration.yml:
############################################## 
input_ids: 
  user: ./tests/input_pulse_file
output_ids:
  database: GITLAB
  user: ./tests/input_pulse_file/
  shot: 114100
  run: 42
expected_ids:
  database: GITLAB
  user: ./tests/input_pulse_file/
  shot: 114100
  run: 43

For your convenience, the Actor Testing Framework comes with a custom imasdb, located under: testing/input_pulse_file/GITLAB directory path that must be accessed from the relative path of the project root directory.

CI script

 Repository side CI scripts

GITLAB has a convention to put CI script in the root directory. Actor Testing Framework follows this convention, but can be extended with an additional file in the root directory: .gitlab-ci.yml. Then, in this upper CI script, you need to execute the lower file: tests/ci_scripts/.gitlab-ci.yml.

Project_root
├── .gitlab-ci.yml 	# <-- Upper CI script
│
├── root_files.extension
├── root_directiories/
│
└── tests/
 	└── ci_scripts/
  		└── .gitlab-ci.yml	# <-- Lower CI script 

##############################################
- example: inside upper ci script: .gitlab-ci.yml
############################################## 
include:
  - 'tests/ci_scripts/.gitlab-ci.yml'

 Building Actor Native Codes

Inside the Actor Testing Framework test directory there is a dedicated directory for building native actor code instructions called: native_codes_build/. By default, there is a build_actor.sh script that should provide step-by-step instructions on how to build an actor. For example, it could be a simple make command, but first you need to change the directory your codes are in, basically it should be done the same way you build the codes from the root directory.

tests
├── native_codes_build
│   └── build_actor.sh	# <-- This file
.
.
.

 Run Actor

Inside the Actor Testing Framework test directory there is a dedicated directory to run the actor code, usually it will be a workflow, called main_code_run/. By default, there is a run_actor.sh script that should provide step-by-step instructions for starting an actor or executing a workflow. Basically, it should reflect the same way you run actor from the root directory.

tests 
├── main_code_run
│   └── run_actor.sh	# <-- This file
.
.
.

 Test Cases - Pytest

Actor Testing Framework runs tests using the Python Pytest framework. This requires the user to have a basic understanding of the Pytest framework. All query related scripts are placed in the tests/pytest_scripts directory. The user can provide as many test cases as needed and is responsible for writing them.

tests
├── pytest_scripts
│   ├── conftest.py
│   ├── __init__.py
│   ├── pytest.ini
│   └── test_actor.py
.
.
.

 Example

The Actor Testing Framework includes an example/ directory that holds an example of using a framework called the eq2eq fake actor that simply rewrites one IDS to another.

Project root
|-- example	# <-- This directory
|   |-- .gitlab-ci.yml
|   |-- actor
|   |-- native-codes
|   |-- tests
|   `-- workflows

To check it, just take out the example directory to a separate repository and try doing CI tests. By default, the example actor operates on pulsefiles located in input_pulse_file/ directory, located under example/tests.

This work has been carried out within the framework of the EUROfusion Consortium, funded by the European Union via the Euratom Research and Training Programme (Grant Agreement No. 101052200—EUROfusion).  Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those  of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. The project activities are co-financed by Polish Ministry of Science and Higher Education from financial resources of the programme entitled "PMW" in 2021, 2022.