Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Installing JOREK on Eurofusion gateway.

Part of the instructions is copied from JOREK wiki pages, eventually instructions found here should overlap with those from JOREK wiki.

How to obtain JOREK source code - ITER git platform

  • Request an account for the ITER platform via email.
  • Request access to JOREK code
  • Create an ssh keypair (help)
    • Run the command

      Code Block
      languageactionscript3
      ssh-keygen -t rsa -C "<key identifier>"

      Example:

      Code Block
      languageactionscript3
      ssh-keygen -t rsa -C "git.iter.org"


    • You will be prompted for a location to save your key. Be careful not to overwrite any existing keys that you still need!

    • You can add a password for an additional layer of security (encrypting your private key file) or proceed without a password. Push Enter when prompted for a password if you don't want to add a password.

    • Upload your public ssh key: https://git.iter.org/plugins/servlet/ssh/account/keys

      • The public key is stored in the .pub file.

      • The .pub file is a regular text file. You will need to copy its contents into the web form that you get after clicking “Add key”.

    • If you are using other keys, configure a .ssh/config file by adding:

      Code Block
      Host git.iter.org
       HostName git.iter.org
       IdentityFile ~/.ssh/id_rsa_iter

      if you named your file id_rsa_iter,

  • Get the code from ITER platform

Your home directory is located on AFS  file system (/afs/eufus.eu/user/g/<username>) which is protected by ACL and by default is not accessible by batch. If you intend to submit your jobs to batch system it is better to use GPFS file system. Save your repository for example in /pfs/work/<username>/jorek folder. You can read more about Gateway Storage Areas here.

    • Clone JOREK repository

      Code Block
      git clone ssh://git@git.iter.org/stab/jorek.git


Compiling JOREK

  • The following modules need to be loaded e.g. by including in the ~/.bashrc 

    Code Block
    module purge
    module load cineca
    module load intel/pe-xe-2017--binary
    module load intelmpi/2017--binary
    module load mkl/2017--binary
    module load fftw/3.3.4--intelmpi--2017--binary
    module load szip/2.1--gnu--6.1.0
    module load zlib/1.2.8--gnu--6.1.0
    module load hdf5/1.8.17--intelmpi--2017--binary


  • Before compilation one has to specify which compiler and libraries will be used. This depends on the machine. For the Eurofusion Gateway the following Makefile.inc for compiling JOREK needs to be copied to JOREK repository directory 

    Code Block
    titleMakefile.inc
    linenumberstrue
    collapsetrue
     --- Select physics model
    MODEL             = model303
    
    # --- Compiler and options
    FC                = mpiifort
    CC                = mpiicc
    CXX               = mpiicpc
    
    FFLAGS += -O3
    FFLAGS += -vecabi=compat -mcmodel=medium
    
    # --- Various switches
    USE_PASTIX              = 1
    USE_PASTIX_MURGE        = 0
    USE_COMPLEX_PRECOND     = 0
    USE_STRUMPACK           = 0
    USE_HDF5                = 1
    USE_FFTW                = 1
    USE_MUMPS               = 0
    USE_DIRECT_CONSTRUCTION = 0
    USE_BLOCK               = 1
    
    LIBDIR = /afs/eufus.eu/user/g/g2iholod/public
    LIBS += -liomp5 -pthread -ldl -lm
    
    ifeq (1, $(USE_MUMPS))
    # --- MUMPS
     MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
             -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
     MKLINC = -I${MKL_HOME}/include
    
     MUMPSDIR = $(LIBDIR)/MUMPS_5.2.1
     LIB_MUMPS = -L$(MUMPSDIR)/lib -ldmumps -ldmumps -lmumps_common -lpord
     INC_MUMPS = $(MUMPSDIR)/include
    
     LIB_MUMPS += $(MKLLIB)
     INC_MUMPS += $(MKLINC)
    endif
    
    ##################
    
    ifeq (1, $(USE_PASTIX))
     MKLLIB = -L${MKL_HOME}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
     MKLINC = -I${MKL_HOME}/include
    
     INC_PASTIX += $(MKLINC)
     LIB_PASTIX += $(MKLLIB)
    
     PASTIX_DIR=$(LIBDIR)/pastix_5.2.3/install
     SCOTCH_DIR=$(LIBDIR)/scotch_5.1.12
    
     LIB_PASTIX += $(PASTIX_DIR)/libpastix.a $(SCOTCH_DIR)/lib/libscotch.a $(SCOTCH_DIR)/lib/libscotcherr.a
     INC_PASTIX += -I$(PASTIX_DIR)
    
    endif
    
    ##################
     
    ifeq (1, $(USE_STRUMPACK))
     MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
             -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
     MKLINC = -I${MKL_HOME}/include
     
     PARMETIS_HOME=$(LIBDIR)/parmetis-4.0.3/build
     STRUMPACK_HOME=$(LIBDIR)/STRUMPACK_MKL/install
     
     STRUMPACKINC = -I$(STRUMPACK_HOME)/include
     STRUMPACKLIB = $(STRUMPACK_HOME)/lib/libstrumpack.a
     
     STRUMPACKINC += -I$(PARMETIS_HOME)/include
     STRUMPACKLIB += $(PARMETIS_HOME)/libparmetis/libparmetis.a
     
     STRUMPACKINC += -I$(PARMETIS_HOME)/metis/include
     STRUMPACKLIB += $(PARMETIS_HOME)/libmetis/libmetis.a
     
     STRUMPACKINC += $(MKLINC)
     STRUMPACKLIB += $(MKLLIB)
     
     DEFINES += -DNEWSPK
    endif
     
    ##################
     
    LIBFFTW           = $(FFTW_LIB)/libfftw3f.a
    INC_FFTW          = -I$(FFTW_INC)
     
    HDF5INCLUDE = $(HDF5_HOME)/include
    HDF5LIB     =-L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -L$(SZIP_LIB) -lsz -lz

    The above Makefile.inc can be modified. In line 2 the model is defined. There are several models available 199, 303, 401, 500, 501, 502, 600, 710, 711 and 712. The list of input parameters for each model can be found here. For the demonstration purposes model 303 is chosen.

    Most of the necessary libraries are already available at the Gateway. There are two libraries Pastix (5.2.3) and Scotch (5.1.12) which are not available and have to be provided separately. The precompiled libraries can be found in ~g2iholod/public folder. The apropriate path is set in line 23 of Makefile.inc. Depending on the setting MUMPS library may also be needed but it is not used in the example run (see line19 of Makefile.inc).
  • Compile JOREK with 

    Code Block
    make -j 8

    This creates an executable file "jorek_model<model_number>" which is "jorek_model303" in our case.

Submitting jobs

Example of the batch job submission script

Code Block
languagebash
titlesubmit.sh
#!/bin/bash

#SBATCH --job-name=jorek
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=24
#SBATCH --time=00:10:00
#SBATCH --partition=gw

export OMP_NUM_THREADS=24

srun /pfs/work/$USER/jorek/jorek_model303 < ./inxflow > logfile.out

The "inxflow" file with example input parameters can be found in ~g2iholod/public/jorek_dev/tmp directory. Since the example run is expected to finished within 30 minutes it is possible to change the standard partition "gw" to the debugging partition "gwdbg" by setting

Code Block
languagebash
#SBATCH --partition=gwdbg

in submit.sh script and omit the queue for jobs that take more time.

It is advised to run the script in a different folder than the repository. Output files are saved in a working directory and running jorek in the repository results in mixing source files with input and output files.

Submit the bash script with the command

Code Block
languageactionscript3
sbatch  submit.sh

Check your submission status with 

Code Block
squeue -u $USER

Creating namelist

Procedure of creating a JOREK namelist for a given model number is a variation of running a "JOREK for the first time case" instruction described in jorek wiki. We describe the procedure step by step here.

It is reccomended that you have access to JOREK repository. This way, you are sure that you are using the latest version of the code and namelist. Furthermore, it allows you to test for different models easily yourself (the namelists of the models are not exactly identical).


Take the very simple intear test case from the tutorial and run it for model199 with 0 time steps (nstep=0):

===== - Running a simple test case =====

We will run a very simple test case: a **tearing mode in a large aspect-ratio circular plasma**. You will have to find your own way of how to structure directories for your simulations. For the prupose of this instruction we assume you have created the following directory structure with the source code and the ''Makefile.inc'' prepared according to the instructions section in 1.2:

Code Block
  jorek-namelist/
    jorek.git/                  # the source code
    intear_ntor3/               # run directory

If you want to clone the repository to a specific folder,  'jorek.git' in our example, you can do it by adding 'folder.git' to 'git clone ...' command (see Section 1.1):

Code Block
cd jorek-namelist/
git clone ssh://git@git.iter.org/stab/jorek.git jorek.git


To write out the namelist, you need to add right before the endif in line 193 of in models/model199/initialise_parameters.f90 the following three lines:

   open(42, file='out.nml', status='replace', action='write')

   write(42,in1)

   close(42)

Same for the other models although the line number may change.


First, we compile the code with appropriate settings for the test case:

Code Block
  cd jorek.git/
  ./util/config.sh model=199 n_tor=3 n_period=1 n_plane=4     # run with toroidal mode numbers n=0,1
  make clean
  make -j 8
  make -j 8 jorek2vtk jorek2_postproc


Then, we copy the compiled binaries and the input file for the test case into the run directory (the input file ''intear'' contains the information necessary to run our test case, more information about the namelist input files is given further below):

Code Block
  cp jorek_model199 jorek2vtk jorek2_postproc namelist/model199/intear ../intear_ntor3
  cd ../intear_ntor3


Now, open with your favorite editor the file ''intear'', which contains the description of the test case to be simulated. Many things here will not immediately be self-explanatory of course. The only thing we do now is to change the settings for the time stepping by setting

Code Block
  tstep = 3000.
  nstep = 0


Important notice: You can run the other models using the same intear input file as well. However, it is not guaranteed that it works for all the models.


Managing Jorek parameters using HDF5

Presentation of the tool

A new tool written in C has been developed to manage Jorek namelist parameters files using HDF5.

The tool has been presented in May 2022 to the Jorek team at the Max Planck Institute.

Here are the slides:


View file
nameFleury_Jorek_namelist_converter_final.pdf
height250


Compiling and using the tool from the command line


The tool can be currently executed from the command line. Soon, it will be integrated to the Jorek fortran code in order to be called from Jorek in different use-cases.


COMPILING THE SOURCES

Code Block
$ cd jorek/util/nmlist2h5

Requirements:
HDF5 version >= 1.12

To use the provided Makefile, be sure that:
- The HDF5_HOME environment variable is defined in order to find the HDF5 header files
- The HDF5 lib directory is referenced by the LD_LIBRARY_PATH variable, for linking
- BOOST library is available in your system

Compiling:
$ make -f Makefile_namelist

Example:
[nmlist2h5]$ make -f Makefile_namelist
gcc -c -o namelist2h5.o namelist2h5.c
gcc -c -o h5_2_namelist.o h5_2_namelist.c
gcc -c -o h5_utils.o h5_utils.c
gcc -c -o namelist_api.o namelist_api.c
g++ -pthread -Wl,--no-undefined -o namelist_api namelist2h5.o h5_2_namelist.o h5_utils.o namelist_api.o -L/Applications/libraries/hdf5/1.12.0/gcc/6.4.0/lib -lhdf5

 API

Code Block
Namelist to HDF5 Converter API:
//---------------------------------------------------------------------
int new_simulation(const char* h5_namelist_folder, const char* nml_file_name);
int restart(const char* h5_namelist_folder, const char* nml_file_name, const char* jorek_restart_file_name, const char* time_index);
int write_restart_file (const char* h5_namelist_folder, const char* jorek_restart_file);
int extract_from_h5_namelist (const char* h5_namelist_folder, const char* nml_file, const char* time_index);
int extract_from_jorek_restart_file (const char* h5_namelist_folder, const char* jorek_restart_file_name, const char* nml_file_name, const char* time_index);
//---------------------------------------------------------------------


NEW SIMULATION:

int new_simulation(const char* h5_namelist_folder, const char* nml_file_name);

/** Creates a new HDF5 namelist file from an existing JOREK parameters file in CSV format. Parameters are written in the '00000' HDF5 group which will be created.
* Inputs:
* h5_namelist_folder (string) : directory path in which the new HDF5 namelist file will be created
* nml_file_name (string) : name of the JOREK namelist file in CSV format, the file must be supplied in the 'h5_namelist_folder', an error will occur otherwise.
* Returns:
* 0 if the operation is successfull or a code status < 0 otherwise.
**/


RESTART:

int restart(const char* h5_namelist_folder, const char* nml_file_name, const char* jorek_restart_file_name, const char* time_index);

/** If the HDF5 namelist file already exists in the folder specified by 'h5_namelist_folder', 'restart' will update this file from the specified JOREK restart file 'jorek_restart_file_name':
* - all HDF5 groups YYYYY >= time_index=XXXXX are removed from the HDF5 namelist file
* - the parameters from the file 'nml_file_name' (in CSV format) are added to the HDF5 namelist file in the HDF5 group XXXXX
* If the HDF5 namelist does not exist in the folder specified by 'h5_namelist_folder', 'restart' will create a new HDF5 namelist file in the folder 'h5_namelist_folder':
* - all parameters of the JOREK restart file specified by 'jorek_restart_file_name' are added to the HDF5 namelist file
* - the parameters located in the file 'nml_file_name' (in CSV format) are added to the HDF5 namelist file in the HDF5 group time_index=XXXXX
* Inputs:
* h5_namelist_folder (string) : directory path in which the HDF5 file is located. If the file does not exist, it will be created in this directory.
* nml_file_name (string) : name of the JOREK parameters file in CSV format, the file must be supplied in the 'h5_namelist_folder', an error will occur otherwise.
* jorek_restart_file_name (string) : name of the JOREK restart file in HDF5 format, the file must be supplied in the 'h5_namelist_folder' if the HDF5 namelist file does not exist, an error will occur otherwise.
* time_index (string) : name of the HDF5 group of the HDF5 namelist file where the namelist parameters from the file 'nml_file_name' will be added
* Returns:
* 0 if the operation is successfull or a code status < 0 otherwise.
**/


WRITE RESTART FILE:

/** Creates a new HDF5 JOREK restart file. All parameters from the existing HDF5 namelist file are written in the 'namelist' HDF5 group.
* If the HDF5 namelist file does not exist in the specified 'h5_namelist_folder' directory, an error will occur.
* Inputs:
* h5_namelist_folder (string) : directory path in which the new HDF5 JOREK restart file will be created
* jorek_restart_file_name (string) : name of the JOREK restart file in HDF5 format
* Returns:
* 0 if the operation is successfull or a code status < 0 otherwise.
**/

EXTRACT FROM H5 NAMELIST:

/** Creates a new namelist parameters file in CSV format. All parameters from the supplied HDF5 namelist file are extracted from the HDF5 group 'time_index' and copied to the CSV namelist parameters file.
* Inputs:
* h5_namelist_folder (string) : directory path in which the CSV namelist file will be created. This directory should contain the supplied HDF5 namelist file, an error will occur otherwise.
* nml_file_name (string) : name of the CSV namelist file to be created
* time_index (string) : time index with format 'XXXXX'
* Returns:
* 0 if the operation is successfull or a code status < 0 otherwise.
**/

EXTRACT FROM JOREK RESTART FILE:

/** Creates a new namelist parameters file in CSV format. All parameters from the existing JOREK HDF5 restart file are extracted from the HDF5 group 'namelist/time_index' and copied to the CSV namelist file.
* Inputs:
* h5_namelist_folder (string) : directory path in which the CSV namelist file will be created. This directory should contain the supplied HDF5 namelist file, an error will occur otherwise.
* jorek_restart_file_name (string) : name of the JOREK restart file
* nml_file_name (string) : name of the CSV namelist file to be created
* time_index (string) : time index with format 'XXXXX'
* Returns:
* 0 if the operation is successfull or a code status < 0 otherwise.
**/

EXAMPLES

Code Block
NEW SIMULATION

./namelist_api new_simulation /Home/LF218007/jorek out711.nml
analysing file, 13453 line(s) found with identifier, (n_line=15864)
File analysis ended.
Number of identifiers=13453.
Number of parameters=301.
HDF5 dataset for parameter GRID_TO_WALL already previously defined, ignoring new definition
HDF5 dataset for parameter FIX_AXIS_NODES already previously defined, ignoring new definition


Checking the file with h5dump:

h5dump namelist.h5 |less

HDF5 "namelist.h5" {
GROUP "/" {
GROUP "00000" {
DATASET "ADAPTIVE_TIME" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "F
"
}
}
DATASET "AKI_NEO_CONST" {
…


RESTART

./namelist_api restart /Home/LF218007/jorek out711.nml jorek_restart.h5 00001
copying time index: 00001...
analysing file, 13453 line(s) found with identifier, (n_line=15864)
File analysis ended.
Number of identifiers=13453.
Number of parameters=301.
HDF5 dataset for parameter GRID_TO_WALL already previously defined, ignoring new definition
HDF5 dataset for parameter FIX_AXIS_NODES already previously defined, ignoring new definition

Checking the file with h5ls:

$ h5ls namelist.h5
00000 Group
00001 Group


WRITE RESTART FILE:

./namelist_api write_restart_file /Home/LF218007/jorek/jorek_repo/jorek/util jorek_restart.h5


Checking the file with h5ls:

h5ls -r jorek_restart.h5

/ Group
/namelist Group
/namelist/00000 Group
/namelist/00000/ADAPTIVE_TIME Dataset {SCALAR}
/namelist/00000/AKI_NEO_CONST Dataset {SCALAR}
/namelist/00000/AMIN Dataset {SCALAR}
/namelist/00000/AMIX Dataset {SCALAR}
...
/namelist/00001 Group
/namelist/00001/ADAPTIVE_TIME Dataset {SCALAR}
/namelist/00001/AKI_NEO_CONST Dataset {SCALAR}
/namelist/00001/AMIN Dataset {SCALAR}
/namelist/00001/AMIX Dataset {SCALAR}
...
/namelist/00001/Z_LIMITER Dataset {1000/Inf}


EXTRACT FROM H5 NAMELIST:

./namelist_api extract_from_h5_namelist /Home/LF218007/jorek test.nml 00001
found group 00001
matched group 00001
Converting HDF5 file /Home/LF218007/jorek/jorek_repo/jorek/util/namelist.h5 ...


more test.nml

&IN1
ADAPTIVE_TIME = F
,
AKI_NEO_CONST = 0.000000e+00,
AMIN = 1.000000e+00,
AMIX = 0.000000e+00,
AMIX_FREEB = 8.500000e-01,
AMU_NEO_CONST = 0.000000e+00,
AUTODISTRIBUTE_MODES = T
,
…


EXTRACT FROM JOREK RESTART FILE:

./namelist_api extract_from_jorek_restart_file /Home/LF218007/jorek jorek_restart.h5 test2.nml 00001
found group 00001
matched group 00001
Converting HDF5 file /Home/LF218007/jorek/jorek_repo/jorek/util/jorek_restart.h5 ...


more test2.nml

&IN1
ADAPTIVE_TIME = F
,
AKI_NEO_CONST = 0.000000e+00,
AMIN = 1.000000e+00,
AMIX = 0.000000e+00,
AMIX_FREEB = 8.500000e-01,
AMU_NEO_CONST = 0.000000e+00,
AUTODISTRIBUTE_MODES = T
,
…