...
The following modules need to be loaded e.g. by including in the ~/.bashrc
Code Block module purge module load cineca module load intel/pe-xe-2017--binary module load intelmpi/2017--binary module load mkl/2017--binary module load fftw/3.3.4--intelmpi--2017--binary module load szip/2.1--gnu--6.1.0 module load zlib/1.2.8--gnu--6.1.0 module load hdf5/1.8.17--intelmpi--2017--binary
Before compilation one has to specify which compiler and libraries will be used. This depends on the machine. For the Eurofusion Gateway the following Makefile.inc for compiling JOREK needs to be copied to JOREK repository directory
Code Block title Makefile.inc linenumbers true collapse true --- Select physics model MODEL = model303 # --- Compiler and options FC = mpiifort CC = mpiicc CXX = mpiicpc FFLAGS += -O3 FFLAGS += -vecabi=compat -mcmodel=medium # --- Various switches USE_PASTIX = 1 USE_PASTIX_MURGE = 0 USE_COMPLEX_PRECOND = 0 USE_STRUMPACK = 0 USE_HDF5 = 1 USE_FFTW = 1 USE_MUMPS = 0 USE_DIRECT_CONSTRUCTION = 0 USE_BLOCK = 1 LIBDIR = /afs/eufus.eu/user/g/g2iholod/public LIBS += -liomp5 -pthread -ldl -lm ifeq (1, $(USE_MUMPS)) # --- MUMPS MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \ -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 MKLINC = -I${MKL_HOME}/include MUMPSDIR = $(LIBDIR)/MUMPS_5.2.1 LIB_MUMPS = -L$(MUMPSDIR)/lib -ldmumps -ldmumps -lmumps_common -lpord INC_MUMPS = $(MUMPSDIR)/include LIB_MUMPS += $(MKLLIB) INC_MUMPS += $(MKLINC) endif ################## ifeq (1, $(USE_PASTIX)) MKLLIB = -L${MKL_HOME}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core MKLINC = -I${MKL_HOME}/include INC_PASTIX += $(MKLINC) LIB_PASTIX += $(MKLLIB) PASTIX_DIR=$(LIBDIR)/pastix_5.2.3/install SCOTCH_DIR=$(LIBDIR)/scotch_5.1.12 LIB_PASTIX += $(PASTIX_DIR)/libpastix.a $(SCOTCH_DIR)/lib/libscotch.a $(SCOTCH_DIR)/lib/libscotcherr.a INC_PASTIX += -I$(PASTIX_DIR) endif ################## ifeq (1, $(USE_STRUMPACK)) MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \ -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 MKLINC = -I${MKL_HOME}/include PARMETIS_HOME=$(LIBDIR)/parmetis-4.0.3/build STRUMPACK_HOME=$(LIBDIR)/STRUMPACK_MKL/install STRUMPACKINC = -I$(STRUMPACK_HOME)/include STRUMPACKLIB = $(STRUMPACK_HOME)/lib/libstrumpack.a STRUMPACKINC += -I$(PARMETIS_HOME)/include STRUMPACKLIB += $(PARMETIS_HOME)/libparmetis/libparmetis.a STRUMPACKINC += -I$(PARMETIS_HOME)/metis/include STRUMPACKLIB += $(PARMETIS_HOME)/libmetis/libmetis.a STRUMPACKINC += $(MKLINC) STRUMPACKLIB += $(MKLLIB) DEFINES += -DNEWSPK endif ################## LIBFFTW = $(FFTW_LIB)/libfftw3f.a INC_FFTW = -I$(FFTW_INC) HDF5INCLUDE = $(HDF5_HOME)/include HDF5LIB =-L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -L$(SZIP_LIB) -lsz -lz
The above Makefile.inc can be modified. In line 2 the model is defined. There are several models available 199, 303, 401, 500, 501, 502, 600, 710, 711 and 712. The list of input parameters for each model can be found here. For the demonstration purposes model 303 is chosen.
Most of the necessary libraries are already available at the Gateway. There are two libraries Pastix (5.2.3) and Scotch (5.1.12) which are not available and have to be provided separately. They The precompiled libraries can be found in ~g2iholod/public folder. The apropriate path is set in line 23 of Makefile.inc. Depending on the setting MUMPS library may also be needed but it is not used in the example run (see line line19 of Makefile.inc).Compile JOREK with
Code Block make -j 8
...
Example of the batch job submission script script
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bash #SBATCH --job-name=jorek #SBATCH --nodes=1 #SBATCH --ntasks=2 #SBATCH --cpus-per-task=24 #SBATCH --time=00:10:00 #SBATCH --partition=gw export OMP_NUM_THREADS=24 srun ./pfs/work/$USER/jorek/jorek_model303 < ./inxflow > logfile.out |
The "inxflow" file with example input parameters can be found in ~g2iholod/public/jorek_dev/tmp directory. Since the example run is expected to finished within 30 minutes it is possible to change "debugging partition" gwdbg by setting
Code Block | ||
---|---|---|
| ||
#SBATCH --partition=gwdbg |
in submit.sh script and omit the queue for jobs that take more time.
It is advised to run the script in a different folder than the repository. Output files are saved in a working directory and running jorek in repository results in mixing source files with inout and output.
Submit the bash script with the command
Code Block | ||
---|---|---|
| ||
sbatch submit.sh |
Check your submission status with
Code Block |
---|
squeue -u $USER |