At the moment PyAL
is used across all the Python
based actors (generated via FC2K
). PyAL
provides another layer between Access Layer
and actors.
Event though current solution works it has some flaws:
Python
template (FC2K
) is composed of one, large file that is hard to maintainPyAL
module (as a middle layer) seems to be obsolete now. As we plan to push everything towards IMAS
based approach it seems reasonable to put PyAL
features directly into Python
interface, or as an external module that is quite close to Python
HLI
. At the moment, PyAL
is hard to maintain due to:
PyAL
serves as yet another level in the whole structure and adds unnecessary complexity to the solutionPyAL
using features coming purely from Python HLI
PyAL
doesn't support IDS
description based data transfer - it means we have to read/write data between calls to native routinesPyAL
is a quite hard process. Lots of thing are tangled together via underlaying objects. It means that changing code in one place affects other parts.Workflows are the element that will probably require least changes. However it still can be improved.
IMAS
based Python HLI
... from pyal import ALEnv, IDS, IDSRef ! features from pyal module ... import mpi_actors ! this is one is responsible for saving input.txt, running mpi code and loading output.txt back # only input arguments, outputs are returned (as a list if more than 1) # kwargs for additional arguments # TODO what about code parameters? ! maybe we can improve this part a little bit. Maybe we can pass arguments into function ! instead of passing dictionary def ${actor}( <#if arguments??> <#list arguments as arg> <#if arg.in>${arg.name},</#if> </#list> </#if> <#if hasCodeParams>codeparams=None, </#if> exec_type='${default_exec_type}', **kwargs): ... ... ids_tmp = ALEnv().ids_tmp ! temporary database ... ... ... ... arg['cval'] = IDSRef() ! IDS class (wrapper provided by PyAL) # the important fieds # we use the temporary shot/run arg['cval'].idx = ids_tmp.expIdx ! we are accessing temporary db # TODO can we get the IDS name from the ids object? # so that fc2k does not have to fill this in the template arg['cval'].ids = '${arg.idsname}' # the accurrence is just for temporary storage # conflicts will be resolved later arg['cval'].occurrence = 0 # these fields are redundant arg['cval'].shot = ids_tmp.shot ! we are accessing temporary db arg['cval'].run = ids_tmp.run ! we are accessing temporary db arg['cval'].machine = ids_tmp.machine ! we are accessing temporary db arg['cval'].user = ids_tmp.user ! we are accessing temporary db arg['cval'].version = ids_tmp.version ! we are accessing temporary db ... ... ... # check conflicting occurences and store data occ_dict = {} for arg in arguments_dict.values(): if isinstance(arg['cval'], IDSRef): ! IDS wrapper provided by PyAL occ_dict[arg['cval'].ids] = 1 + occ_dict.get(arg['cval'].ids, -1) arg['cval'].occurrence = occ_dict[arg['cval'].ids] # store input data if arg['in']: ids_tmp.put(arg['value'], arg['cval'].occurrence, single_slice=True) ! we are accessing temporary db ... ... ... <#if isTimeSliceActor> # adding timein, timeout and iter is_time_slice_actor = True timein = None for arg_name in arguments_order: arg = arguments_dict[arg_name] if isinstance(arg['cval'], IDSRef) and arg['in']: ! IDS wrapper provided by PyAL if timein is None: if len(arg['value'].time): timein = float(arg['value'].time) elif timein != arg['value'].time: logger_${actor}.warning('time %g of %s is different from the first ids, using %g' % (arg['value'].time, arg_name, timein)) ... ... ... ! MPI based execution if is_time_slice_actor: mpi_out = mpi_actors.mpi_run_actor(_actor_name, _actor_executable, arguments_dict, arguments_order, arguments_order_out, codeparams_str, xsd_file=_xsd_file, default_xml_file=_default_xml_file, mpi_command='mpirun', mpi_params=[('-np', str(mpi_processes))], mpi_env={}, rm_tmp_dir=rm_tmp_dir, time_in=timein, iter_in=iter_) else: mpi_out = mpi_actors.mpi_run_actor(_actor_name, _actor_executable, arguments_dict, arguments_order, arguments_order_out, codeparams_str, xsd_file=_xsd_file, default_xml_file=_default_xml_file, mpi_command='mpirun', mpi_params=[('-np', str(mpi_processes))], mpi_env={}, rm_tmp_dir=rm_tmp_dir) ... ... ... # get output data results = [] for arg_name in arguments_order_out: arg = arguments_dict[arg_name] if isinstance(arg['cval'], IDSRef): ! IDS wrapper provided by PyAL # read IDS data if kwargs.get('get_output_idss', True): if is_time_slice_actor: ! we are accessing temporary db results.append(ids_tmp.get(arg['cval'].ids, arg['cval'].occurrence, time=timeout)) else: results.append(ids_tmp.get(arg['cval'].ids, arg['cval'].occurrence)) ... ... ... |