Python profiling

1. Table of Contents

2. Profilers

I use two profilers:

cProfile - https://docs.python.org/3/library/profile.html

line-profiler - https://pypi.org/project/line-profiler/

3. requirements

python 3.10 module load itm-python/3.10

line-profiler tapip install line-profiler

4. How to run?

prepare a standard launch
Copy python folder from /gss_efgw_work/work/g2fjc/jintrac/v220922/python to some location on pfs for example: /pfs/work/g2pbloch/python
Change JINTRAC_PYTHON_DIR in rjettov (430 line) to python folder on pfs from step 2
Change run_python_driver (line 49) in /pfs/work/g2pbloch/python :
cProfile : mpirun --allow-run-as-root -np $NPROC python -u -m cProfile -o jintrac.prof /pfs/work/g2pbloch/jetto_profiler/jintrac_imas_driver.py mpi
line-profiler : mpirun --allow-run-as-root -np $NPROC python -u -m kernprof -l /pfs/work/g2pbloch/jetto_profiler/jintrac_imas_driver.py mpi
When we use line-profiler we must add wrapper to profiling function. In this case we should add @profile upper jintrac_imas_driver function in jintrac_imas_driver.py :
function wrapper
```
@profile
def jintrac_imas_driver(params, components, mpi='no'):

    """JINTRAC-IMAS generic workflow driver."""
```
Run ./rjettov -S -I -p -x64 test v220922 g2fjc

5. cProfile analysis

To read data from test.prof we need python script. We can use e.g.

cprofile_script.py

import pstats
stats = pstats.Stats('test.prof')
stats.strip_dirs().sort_stats(pstats.SortKey.TIME).print_stats()

This script print data sorted by tottime column ( In tottime column is total time spent in the given function (and excluding time made in calls to sub-functions)). These are the first few lines of the results :

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        3   81.304   27.101   81.304   27.101 {imas_3_37_0_ual_4_11_0._ual_lowlevel.ual_open_pulse}
  2891220   42.947    0.000   52.709    0.000 {imas_3_37_0_ual_4_11_0._ual_lowlevel.ual_read_data_array}
  2497358   16.127    0.000   16.127    0.000 {imas_3_37_0_ual_4_11_0._ual_lowlevel.ual_read_data_scalar}
  2899948    2.784    0.000    2.784    0.000 {built-in method numpy.zeros}
   815194    2.391    0.000    2.391    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   427578    1.225    0.000    2.589    0.000 fromnumeric.py:38(_wrapit)
   815196    1.015    0.000    7.988    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
   387616    0.876    0.000    2.809    0.000 fromnumeric.py:69(_wrapreduction)
   427578    0.848    0.000    0.848    0.000 {method 'reshape' of 'numpy.ndarray' objects}
   427578    0.685    0.000    3.358    0.000 fromnumeric.py:51(_wrapfunc)
   387785    0.684    0.000    0.684    0.000 {method 'items' of 'dict' objects}

We can also make an image from data:

Install gprof2dot : pip install gprof2dot
run : python -m gprof2dot -f pstats test.prof | dot -Tsvg -o output.svg

Example image looks like this:

output.svg

6. Line-profiler analysis

Page tree

Python profiling