1. Installing JOREK on Eurofusion gateway.

Part of the instructions is copied from JOREK wiki pages, eventually instructions found here should overlap with those from JOREK wiki.

1.1. How to obtain JOREK source code - ITER git platform

Request an account for the ITER platform via email.
Request access to JOREK code
Create an ssh keypair (help)
- Run the command
```
ssh-keygen -t rsa -C "<key identifier>"
```
  Example:
```
ssh-keygen -t rsa -C "git.iter.org"
```
- You will be prompted for a location to save your key. Be careful not to overwrite any existing keys that you still need!
- You can add a password for an additional layer of security (encrypting your private key file) or proceed without a password. Push Enter when prompted for a password if you don't want to add a password.
- Upload your public ssh key: https://git.iter.org/plugins/servlet/ssh/account/keys
  - The public key is stored in the .pub file.
  - The .pub file is a regular text file. You will need to copy its contents into the web form that you get after clicking “Add key”.
- If you are using other keys, configure a .ssh/config file by adding:
```
Host git.iter.org
 HostName git.iter.org
 IdentityFile ~/.ssh/id_rsa_iter
```
  if you named your file id_rsa_iter,
Get the code from ITER platform

Your home directory is located on AFS file system (/afs/eufus.eu/user/g/<username>) which is protected by ACL and by default is not accessible by batch. If you intend to submit your jobs to batch system it is better to use GPFS file system. Save your repository for example in /pfs/work/<username>/jorek folder. You can read more about Gateway Storage Areas here.

Clone JOREK repository

git clone ssh://git@git.iter.org/stab/jorek.git

1.2. Compiling JOREK

The following modules need to be loaded e.g. by including in the ~/.bashrc

module purge
module load cineca
module load intel/pe-xe-2017--binary
module load intelmpi/2017--binary
module load mkl/2017--binary
module load fftw/3.3.4--intelmpi--2017--binary
module load szip/2.1--gnu--6.1.0
module load zlib/1.2.8--gnu--6.1.0
module load hdf5/1.8.17--intelmpi--2017--binary

Before compilation one has to specify which compiler and libraries will be used. This depends on the machine. For the Eurofusion Gateway the following Makefile.inc for compiling JOREK needs to be copied to JOREK repository directory

Makefile.inc

 --- Select physics model
MODEL             = model303

# --- Compiler and options
FC                = mpiifort
CC                = mpiicc
CXX               = mpiicpc

FFLAGS += -O3
FFLAGS += -vecabi=compat -mcmodel=medium

# --- Various switches
USE_PASTIX              = 1
USE_PASTIX_MURGE        = 0
USE_COMPLEX_PRECOND     = 0
USE_STRUMPACK           = 0
USE_HDF5                = 1
USE_FFTW                = 1
USE_MUMPS               = 0
USE_DIRECT_CONSTRUCTION = 0
USE_BLOCK               = 1

LIBDIR = /afs/eufus.eu/user/g/g2iholod/public
LIBS += -liomp5 -pthread -ldl -lm

ifeq (1, $(USE_MUMPS))
# --- MUMPS
 MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKL_HOME}/include

 MUMPSDIR = $(LIBDIR)/MUMPS_5.2.1
 LIB_MUMPS = -L$(MUMPSDIR)/lib -ldmumps -ldmumps -lmumps_common -lpord
 INC_MUMPS = $(MUMPSDIR)/include

 LIB_MUMPS += $(MKLLIB)
 INC_MUMPS += $(MKLINC)
endif

##################

ifeq (1, $(USE_PASTIX))
 MKLLIB = -L${MKL_HOME}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
 MKLINC = -I${MKL_HOME}/include

 INC_PASTIX += $(MKLINC)
 LIB_PASTIX += $(MKLLIB)

 PASTIX_DIR=$(LIBDIR)/pastix_5.2.3/install
 SCOTCH_DIR=$(LIBDIR)/scotch_5.1.12

 LIB_PASTIX += $(PASTIX_DIR)/libpastix.a $(SCOTCH_DIR)/lib/libscotch.a $(SCOTCH_DIR)/lib/libscotcherr.a
 INC_PASTIX += -I$(PASTIX_DIR)

endif

##################
 
ifeq (1, $(USE_STRUMPACK))
 MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKL_HOME}/include
 
 PARMETIS_HOME=$(LIBDIR)/parmetis-4.0.3/build
 STRUMPACK_HOME=$(LIBDIR)/STRUMPACK_MKL/install
 
 STRUMPACKINC = -I$(STRUMPACK_HOME)/include
 STRUMPACKLIB = $(STRUMPACK_HOME)/lib/libstrumpack.a
 
 STRUMPACKINC += -I$(PARMETIS_HOME)/include
 STRUMPACKLIB += $(PARMETIS_HOME)/libparmetis/libparmetis.a
 
 STRUMPACKINC += -I$(PARMETIS_HOME)/metis/include
 STRUMPACKLIB += $(PARMETIS_HOME)/libmetis/libmetis.a
 
 STRUMPACKINC += $(MKLINC)
 STRUMPACKLIB += $(MKLLIB)
 
 DEFINES += -DNEWSPK
endif
 
##################
 
LIBFFTW           = $(FFTW_LIB)/libfftw3f.a
INC_FFTW          = -I$(FFTW_INC)
 
HDF5INCLUDE = $(HDF5_HOME)/include
HDF5LIB     =-L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -L$(SZIP_LIB) -lsz -lz

The above Makefile.inc can be modified. In line 2 the model is defined. There are several models available 199, 303, 401, 500, 501, 502, 600, 710, 711 and 712. The list of input parameters for each model can be found here. For the demonstration purposes model 303 is chosen.

Most of the necessary libraries are already available at the Gateway. There are two libraries Pastix (5.2.3) and Scotch (5.1.12) which are not available and have to be provided separately. The precompiled libraries can be found in ~g2iholod/public folder. The apropriate path is set in line 23 of Makefile.inc. Depending on the setting MUMPS library may also be needed but it is not used in the example run (see line19 of Makefile.inc).

Compile JOREK with
```
make -j 8
```

1.3. Submitting jobs

Example of the batch job submission script

submit.sh

#!/bin/bash

#SBATCH --job-name=jorek
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=24
#SBATCH --time=00:10:00
#SBATCH --partition=gw

export OMP_NUM_THREADS=24

srun /pfs/work/$USER/jorek/jorek_model303 < ./inxflow > logfile.out

The "inxflow" file with example input parameters can be found in ~g2iholod/public/jorek_dev/tmp directory. Since the example run is expected to finished within 30 minutes it is possible to change "debugging partition" gwdbg by setting

#SBATCH --partition=gwdbg

in submit.sh script and omit the queue for jobs that take more time.

It is advised to run the script in a different folder than the repository. Output files are saved in a working directory and running jorek in repository results in mixing source files with inout and output.

Submit the bash script with the command

sbatch  submit.sh

Check your submission status with

squeue -u $USER

Page tree

TSVV-8

1. Installing JOREK on Eurofusion gateway.

1.1. How to obtain JOREK source code - ITER git platform

1.2. Compiling JOREK

1.3. Submitting jobs