At the moment PyAL is used across all the Python based actors (generated via FC2K). PyAL provides another layer between Access Layer and actors.

Current architecture (heavily simplified)

Event though current solution works it has some flaws:

1.1.1.1. Template

  • Python  template (FC2K) is composed of one, large file that is hard to maintain
  • Lots of information between components is passed via dictionaries instead of strictly defined interfaces
  • Logic of the actor is driven by if-else based approach - it's hard to add new features that are not supported at the moment 
  • Lots of hidden assumptions are made in the code (e.g. temporary files creation, arguments that are responsible for script execution, etc.)
  • Error handling is not quite ideal

1.1.1.2. PyAL module 

PyAL module (as a middle layer) seems to be obsolete now. As we plan to push everything towards IMAS based approach it seems reasonable to put PyAL features directly into Python interface,  or as an external module that is quite close to Python HLI. At the moment, PyAL is hard to maintain due to:

  • parts of code being hardcoded (magic values)
  • PyAL serves as yet another level in the whole structure and adds unnecessary complexity to the solution
  • It should be possible to implement features of PyAL using features coming purely from Python HLI
  • PyAL doesn't support IDS description based data transfer - it means we have to read/write data between calls to native routines
  • Changing PyAL is a quite hard process. Lots of thing are tangled together via underlaying objects. It means that changing code in one place affects other parts.

1.1.1.3. Workflow

Workflows are the element that will probably require least changes. However it still can be improved.

  • We can improve the whole solution such way it is based purely on IMAS based Python HLI
  • Once index based (file handler like) exchange  of data workflows will require less time and memory to work - there will be no need of exchanging data back and forth between the calls

1.1.1.4. References to PyAL inside Python based template generated by FC2K

...
from pyal import ALEnv, IDS, IDSRef        ! features from pyal module
...
import mpi_actors                          ! this is one is responsible for saving input.txt, running mpi code and loading output.txt back


# only input arguments, outputs are returned (as a list if more than 1)
# kwargs for additional arguments
# TODO what about code parameters?

! maybe we can improve this part a little bit. Maybe we can pass arguments into function 
! instead of passing dictionary

def ${actor}(
<#if arguments??>
  <#list arguments as arg>
    <#if arg.in>${arg.name},</#if>
  </#list>
</#if>
<#if hasCodeParams>codeparams=None, </#if>
exec_type='${default_exec_type}',
**kwargs):
...
...
    ids_tmp = ALEnv().ids_tmp                                 ! temporary database
...
...
...
...
    arg['cval'] = IDSRef()                                            ! IDS class (wrapper provided by PyAL)
    # the important fieds
    # we use the temporary shot/run
    arg['cval'].idx = ids_tmp.expIdx                                  ! we are accessing temporary db
    # TODO can we get the IDS name from the ids object?
    # so that fc2k does not have to fill this in the template
    arg['cval'].ids = '${arg.idsname}'
    # the accurrence is just for temporary storage
    # conflicts will be resolved later
    arg['cval'].occurrence = 0
    # these fields are redundant
    arg['cval'].shot = ids_tmp.shot                                   ! we are accessing temporary db
    arg['cval'].run = ids_tmp.run                                     ! we are accessing temporary db
    arg['cval'].machine = ids_tmp.machine                             ! we are accessing temporary db
    arg['cval'].user = ids_tmp.user                                   ! we are accessing temporary db
    arg['cval'].version = ids_tmp.version                             ! we are accessing temporary db
...
...
...    
    # check conflicting occurences and store data
    occ_dict = {}
    for arg in arguments_dict.values():
        if isinstance(arg['cval'], IDSRef):                                            ! IDS wrapper provided by PyAL
            occ_dict[arg['cval'].ids] = 1 + occ_dict.get(arg['cval'].ids, -1)
            arg['cval'].occurrence = occ_dict[arg['cval'].ids]
            # store input data
            if arg['in']:
                ids_tmp.put(arg['value'], arg['cval'].occurrence, single_slice=True)   ! we are accessing temporary db
...
...
...
	<#if isTimeSliceActor>
    # adding timein, timeout and iter
    is_time_slice_actor = True
    timein = None
    for arg_name in arguments_order:
        arg = arguments_dict[arg_name]
        if isinstance(arg['cval'], IDSRef) and arg['in']:                              ! IDS wrapper provided by PyAL
            if timein is None:
                if len(arg['value'].time):
                    timein = float(arg['value'].time)
            elif timein != arg['value'].time:
                logger_${actor}.warning('time %g of %s is different from the first ids, using %g' %
                      (arg['value'].time, arg_name, timein))
...
...
...
! MPI based execution

        if is_time_slice_actor:
          mpi_out = mpi_actors.mpi_run_actor(_actor_name, _actor_executable, arguments_dict,
                                           arguments_order, arguments_order_out,
                                           codeparams_str,
                                           xsd_file=_xsd_file,
                                           default_xml_file=_default_xml_file,
                                           mpi_command='mpirun',
                                           mpi_params=[('-np', str(mpi_processes))],
                                           mpi_env={},
                                           rm_tmp_dir=rm_tmp_dir, time_in=timein, iter_in=iter_)
        else:
          mpi_out = mpi_actors.mpi_run_actor(_actor_name, _actor_executable, arguments_dict,
                                           arguments_order, arguments_order_out,
                                           codeparams_str,
                                           xsd_file=_xsd_file,
                                           default_xml_file=_default_xml_file,
                                           mpi_command='mpirun',
                                           mpi_params=[('-np', str(mpi_processes))],
                                           mpi_env={},
                                           rm_tmp_dir=rm_tmp_dir)
...
...
...

    # get output data
    results = []
    for arg_name in arguments_order_out:
        arg = arguments_dict[arg_name]
        if isinstance(arg['cval'], IDSRef):                                          ! IDS wrapper provided by PyAL
            # read IDS data
            if kwargs.get('get_output_idss', True):
                if is_time_slice_actor:                                              ! we are accessing temporary db
                    results.append(ids_tmp.get(arg['cval'].ids, arg['cval'].occurrence, time=timeout))  
                else:
                    results.append(ids_tmp.get(arg['cval'].ids, arg['cval'].occurrence))
...
...
...
  • No labels