You are viewing an old version of this page. View the current version .
Compare with Current
View Page History
« Previous
Version 5
Next »
1. PY-SCIPY crashes during installationSpack has a ready-made recipe for installing the scipy
module for Python, which is called py-scipy
. During IMAS eco-system installation, while providing necessary python packages the one of which is the py-scipy,
the installation of the py-scipy
has crashed. The error message is ambiguous and is shown below:
pip._internal.exceptions.InstallationError: Could not build wheels for scipy, which is required to install pyproject.toml-based projects
1.1. How to reproduce issue Log in to Marconi cluster Load intel/pe-xe-2020–binary
module
module load intel/pe-xe-2020--binary
Load intelmpi/2020--binary
module
module load intelmpi/2020--binary
Clone spack repository
git clone https://github.com/spack/spack.git
To ensure the same environment set the packages.yaml
file as below. This only adds packages marked as external:
Click here to expand...
the packages.yaml
is located here:
vim ./spack/etc/spack/defaults/packages.yaml
And it should have following body:
# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
cmake:
externals:
- spec: cmake@3.23.0
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/cmake/3.23.0/gcc/7.3.0
automake:
externals:
- spec: automake@1.16.5
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/automake/1.16.5/gcc/7.3.0
autoconf:
externals:
- spec: autoconf@2.69
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/autoconf/2.69/gcc/7.3.0
diffutils:
externals:
- spec: diffutils@3.8
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/diffutils/3.8/gcc/7.3.0
perl:
externals:
- spec: perl@5.34.1
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/perl/5.34.1/gcc/7.3.0
tar:
externals:
- spec: tar@1.34
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/tar/1.34/gcc/7.3.0
gettext:
externals:
- spec: gettext@0.21
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/gettext/0.21/gcc/7.3.0
libunwind:
externals:
- spec: libunwind@1.6.2
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/libunwind/1.6.2/gcc/7.3.0
m4:
externals:
- spec: m4@1.4.19
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/m4/1.4.19/gcc/7.3.0
bison:
externals:
- spec: bison@3.8.2
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/bison/3.8.2/gcc/7.3.0
all:
compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
providers:
awk: [gawk]
blas: [openblas, amdblis]
D: [ldc]
daal: [intel-daal]
elf: [elfutils]
fftw-api: [fftw, amdfftw]
flame: [libflame, amdlibflame]
fuse: [libfuse]
gl: [glx, osmesa]
glu: [mesa-glu, openglu]
golang: [go, gcc]
go-external-or-gccgo-bootstrap: [go-bootstrap, gcc]
iconv: [libiconv]
ipp: [intel-ipp]
java: [openjdk, jdk, ibm-java]
jpeg: [libjpeg-turbo, libjpeg]
lapack: [openblas, amdlibflame]
libglx: [mesa+glx, mesa18+glx]
libllvm: [llvm]
libosmesa: [mesa+osmesa, mesa18+osmesa]
lua-lang: [lua, lua-luajit-openresty, lua-luajit]
luajit: [lua-luajit-openresty, lua-luajit]
mariadb-client: [mariadb-c-client, mariadb]
mkl: [intel-mkl]
mpe: [mpe2]
mpi: [openmpi, mpich]
mysql-client: [mysql, mariadb-c-client]
opencl: [pocl]
onedal: [intel-oneapi-dal]
pbs: [openpbs, torque]
pil: [py-pillow]
pkgconfig: [pkgconf, pkg-config]
rpc: [libtirpc]
scalapack: [netlib-scalapack, amdscalapack]
sycl: [hipsycl]
szip: [libaec, libszip]
tbb: [intel-tbb]
unwind: [libunwind]
uuid: [util-linux-uuid, libuuid]
xxd: [xxd-standalone, vim]
yacc: [bison, byacc]
ziglang: [zig]
permissions:
read: world
write: user
Source SPACK to your environment:
source spack/share/spack/setup-env.sh
Install Python:
spack install python@3.9.10 +tkinter %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-matplotlib:
spack install --fail-fast py-matplotlib backend=tkagg %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-cython:
spack install --fail-fast py-cython %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-pyyaml:
spack install --fail-fast py-pyyaml %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-scipy:
spack install --fail-fast py-scipy %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
During this installation you should get an error:
pip._internal.exceptions.InstallationError: Could not build wheels for scipy, which is required to install pyproject.toml-based projects
1.2. SolutionAfter many attempts and tests, a solution was found. The py-scipy
package must be the first Python package to be installed before any other.
Log in to Marconi cluster Load intel/pe-xe-2020–binary
module
module load intel/pe-xe-2020--binary
Load intelmpi/2020--binary
module
module load intelmpi/2020--binary
Clone spack repository
git clone https://github.com/spack/spack.git
To ensure the same environment set the packages.yaml
file as below. This only adds packages marked as external:
Click here to expand...
the packages.yaml
is located here:
vim ./spack/etc/spack/defaults/packages.yaml
And it should have following body:
# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
cmake:
externals:
- spec: cmake@3.23.0
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/cmake/3.23.0/gcc/7.3.0
automake:
externals:
- spec: automake@1.16.5
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/automake/1.16.5/gcc/7.3.0
autoconf:
externals:
- spec: autoconf@2.69
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/autoconf/2.69/gcc/7.3.0
diffutils:
externals:
- spec: diffutils@3.8
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/diffutils/3.8/gcc/7.3.0
perl:
externals:
- spec: perl@5.34.1
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/perl/5.34.1/gcc/7.3.0
tar:
externals:
- spec: tar@1.34
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/tar/1.34/gcc/7.3.0
gettext:
externals:
- spec: gettext@0.21
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/gettext/0.21/gcc/7.3.0
libunwind:
externals:
- spec: libunwind@1.6.2
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/libunwind/1.6.2/gcc/7.3.0
m4:
externals:
- spec: m4@1.4.19
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/m4/1.4.19/gcc/7.3.0
bison:
externals:
- spec: bison@3.8.2
prefix: /marconi/home/userexternal/msmukows/spackdir/libs/bison/3.8.2/gcc/7.3.0
all:
compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
providers:
awk: [gawk]
blas: [openblas, amdblis]
D: [ldc]
daal: [intel-daal]
elf: [elfutils]
fftw-api: [fftw, amdfftw]
flame: [libflame, amdlibflame]
fuse: [libfuse]
gl: [glx, osmesa]
glu: [mesa-glu, openglu]
golang: [go, gcc]
go-external-or-gccgo-bootstrap: [go-bootstrap, gcc]
iconv: [libiconv]
ipp: [intel-ipp]
java: [openjdk, jdk, ibm-java]
jpeg: [libjpeg-turbo, libjpeg]
lapack: [openblas, amdlibflame]
libglx: [mesa+glx, mesa18+glx]
libllvm: [llvm]
libosmesa: [mesa+osmesa, mesa18+osmesa]
lua-lang: [lua, lua-luajit-openresty, lua-luajit]
luajit: [lua-luajit-openresty, lua-luajit]
mariadb-client: [mariadb-c-client, mariadb]
mkl: [intel-mkl]
mpe: [mpe2]
mpi: [openmpi, mpich]
mysql-client: [mysql, mariadb-c-client]
opencl: [pocl]
onedal: [intel-oneapi-dal]
pbs: [openpbs, torque]
pil: [py-pillow]
pkgconfig: [pkgconf, pkg-config]
rpc: [libtirpc]
scalapack: [netlib-scalapack, amdscalapack]
sycl: [hipsycl]
szip: [libaec, libszip]
tbb: [intel-tbb]
unwind: [libunwind]
uuid: [util-linux-uuid, libuuid]
xxd: [xxd-standalone, vim]
yacc: [bison, byacc]
ziglang: [zig]
permissions:
read: world
write: user
Source SPACK to your environment:
source spack/share/spack/setup-env.sh
Install Python:
spack install python@3.9.10 +tkinter %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-scipy:
spack install --fail-fast py-scipy %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-matplotlib:
spack install --fail-fast py-matplotlib backend=tkagg %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-cython:
spack install --fail-fast py-cython %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
Install py-pyyaml:
spack install --fail-fast py-pyyaml %intel@19.1.3.304 cflags="-fPIC" cxxflags="-fPIC" cppflags="-fPIC"
2. BLITZ and missing -pthread
flagAfter installing all dependencies, the very next step is to install IMAS itself. During this installation the cpp interface component is complaining about blitz not being compiled in thread-safe
manner and suggests that maybe the -pthread
flag was missing during Blitz compilation/installation. The bad news is that the whole eco-system was compiled using at least those two flags: '-fPIC -pthread'
, therefore the error is confusing.