Versions Compared

Old Version 5

changes.mady.by.user Bartosz Palak

Saved on

compared with

New Version 6

changes.mady.by.user Bartosz Palak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

RPATH - library discovery at runtime

One of the drawbacks of the module based layout is the problem of pointing to proper library (if we don't have symbolic links with fully qualified names).

This issue can be solved with rpath while compiling the code. Instead of having number of libraries, each with slightly different name, we can use rpath instead.

Code Block
languagebash
gcc -o $@ $< -Iinclude -Llib -lshared -Wl,-rpath,$(CURDIR)/lib

This way, each code will refer to version of library used for linking.

ProsCons
different codes can use different versions of librarieswith each release of new version of the library user's code must be recompiled
each library is released as a separate installation (it's easy to maintain each version separately)it's not possible to move libraries from their original location
we are not forced to build log LD_LIBRARY_PATH variables 
LD_LIBRARY_PATH has lower precedence than rpath 
It is still possible to use LD_PRELOAD to enforce given library to be used (e.g. debugging purposes)

Libraries layout

Examples are provided for a single library (XMLLIB) for the sake of clarity, to be generalised for all libraries.

In case the library depends on the Data Dictionary (DD) version, this DD version information shall be included in the <LIB_VERSION> string.

 

Libraries

The library layout explicitly identifies the library’s name and version and the compiler’s name and version and puts the various files of the library as follows:

Code Block
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/include/*.h
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/include/*.mod
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/lib/*.a
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/lib/*.so
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/lib/pkgconfig/<LIB_NAME>.pc
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/src
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/example
<ROOT_LIB_PATH>/<LIB_NAME>/<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/man

Note that all pc files have the same name, for the sake of simplicity (for the code developer).

In addition, a “central” repository

Code Block
<ROOT_LIB_PATH>/<LIB_NAME>/lib

 contains links to the *.so files (for all library versions and compiler names and versions), in order to keep $LD_LIBRARY_PATH length below the operating system limit.

Code Block
<ROOT_LIB_PATH>/<LIB_NAME>/lib/lib<LIB_NAME>-<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>.so -> ../<LIB_VERSION>-<COMPILER_NAME>-<COMPILER_VERSION>/lib/lib<LIB_NAME>.s

Example

Code Block
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/include/*.h *.mod 
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/lib/*.a *.so
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/lib/pkgconfig/xmllib.pc
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/src
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/example
<ROOT_LIB_PATH>/xmllib/1.0-ifort-17/man
<ROOT_LIB_PATH>/xmllib/lib/libxmllib-1.0-ifort-17.so  -> ../1.0-ifort-17/lib/libxmllib.so

 

Example of XMLLIB installation based on IMAS based release

Code Block
swimas
|-- etc
|   `-- modulefiles
|       `-- xmllib
|           |-- 4.0-gnu-6.1.0
|           `-- 4.0-intel-17.0.4
`-- extra
    `-- xmllib
        |-- 4.0-gnu-4.8.5
        |   |-- include
        |   |   |-- f90_file_reader.mod
        |   |   |-- mod_f90_kind.mod
        |   |   |-- string_manipulation_tools.mod
        |   |   |-- xml2eg_mdl.mod
        |   |   |-- xml_file_reader.mod
        |   |   |-- xmllib_parser.mod
        |   |   |-- xmllib_pathquery.mod
        |   |   `-- xmllib_types.mod
        |   `-- lib
        |       |-- libxmllib.a
        |       `-- pkgconfig
        |           `-- xmllib.pc
        |-- 4.0-intel-17.0.4
        |   |-- include
        |   |   |-- f90_file_reader.mod
        |   |   |-- mod_f90_kind.mod
        |   |   |-- string_manipulation_tools.mod
        |   |   |-- xml2eg_mdl.mod
        |   |   |-- xml_file_reader.mod
        |   |   |-- xmllib_parser.mod
        |   |   |-- xmllib_pathquery.mod
        |   |   `-- xmllib_types.mod
        |   `-- lib
        |       |-- libxmllib.a
        |       `-- pkgconfig
        |           `-- xmllib.pc
        `-- lib                                                                        # this can be completely removed if we start using -rpath
            |-- libxmllib-4.0-gnu-4.8.5.a -> ../4.0-gnu-4.8.5/lib/libxmllib.a          # this can be completely removed if we start using -rpath
            `-- libxmllib-4.0-intel-17.0.4.a -> ../4.0-intel-17.0.4/lib/libxmllib.a    # this can be completely removed if we start using -rpath

RPATH - compiling codes with fixed location of libraries

One of the drawbacks of the module based layout is the problem of pointing to proper library (if we don't have symbolic links with fully qualified names).

This issue can be solved with rpath while compiling the code. Instead of having number of libraries, each with slightly different name, we can use rpath instead.

Actors that consist any workflow can be compiled with different versions of particular library. To allow an actor to resolve all dependent libraries at runtime,

every package of the library, beside providing info about includes (*.h, *.mod) and library (*.so, *.a) has to provide RPATH.

Code Block
languagebash
>> pkg-config --cflags --libs <LIB_NAME>
>> -
Code Block
languagebash
gcc -o $@ $< -Iinclude -Llib -lshared -Wl,-rpath,$(CURDIR)/lib

This way, each code at runtime will refer to version of library used for linking.

ProsCons
different Different codes can use different versions of libraries with each release of new version of the library user's code must be recompiledat runtime

It's not possible to move libraries from their original location

Each each library is released as a separate installation (it's easy to maintain each version separately)it It's not possible to move libraries from their original locationto point to any other (even backward compatible) version
No long we are not forced to build log LD_LIBRARY_PATH variables To use a new release of the library, user's code must be recompiled
LD_LIBRARY_PATH has lower precedence than rpath 
It is still possible to use LD_PRELOAD to enforce given library to be used (e.g. debugging purposes) 

RPATH - slightly different layout of directories

In case of rpath based compilation we don't need centrally located lib directory at all. All files (paths) will be provided by pc files for each and every module.

...

TO BE ADDED

Library directory/module layout

 

...