Generating a Complete ScaLAPACK Library

Procedure

KML_ScaLAPACK contains only some Huawei-developed ScaLAPACK interfaces. To use the KML_ScaLAPACK library, you need to obtain the open source Netlib ScaLAPACK v2.2.0 source package and adapt it to KML_ScaLAPACK to obtain full ScaLAPACK interfaces. The steps are as follows:

Download the Netlib ScaLAPACK v2.2.0 source package. Save the file in a path that can be accessed by the compiler, for example, /openlib/scalapack-2.2.0.tar.gz.
Download the source package at https://github.com/Reference-ScaLAPACK/scalapack/archive/refs/tags/v2.2.0.tar.gz.

Assume that libkscalapack.a is in /HML_LAPACK/build/ins/lib/. Run the following script to compile the original Netlib ScaLAPACK library. To compile Netlib ScaLAPCK, you need to install the Netlib LAPACK library first (refer to Generating a Complete LAPACK Library). For example, if the Netlib LAPACK library is in /openlib/lapack-3.10.1/, then libscalapack_adapt.a will be generated in the scalapack_adapt subdirectory of the current directory.

netlib=/openlib/scalapack.tar.gz
kscalapack=/HML_LAPACK/build/ins/lib/libkscalapack.a
libLapackPath=/openlib/lapack-3.10.1/liblapack.a
libRefBlasPath=/openlib/lapack-3.10.1/libresblas.a

mkdir scalapack_adapt
cd scalapack_adapt

# build netlib scalapack
mkdir netlib
cd netlib
tar zxvf $netlib
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort \
-DLAPACK_LIBRARIES=/$libLapackPath -DBLAS_LIBRARIES=/$libRefBlasPath ../scalapack-2.2.0
make -j
cd ../..

cp netlib/build/lib/libscalapack.a libscalapack_adapt.a

# get symbols defined both in kscalapack and netlib scalapack
nm -g libscalapack_adapt.a | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > netlib.sym
nm -g $kscalapack | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > kscalapack.sym
comm -12 kscalapack.sym netlib.sym > comm.sym 

# update symbols name of libscalapack_adapt.a
while read sym; do \
    if ! nm libscalapack_adapt.a | grep -qe " T ${sym}_\$"; then \
        continue; \
    fi; \
    ar x libscalapack_adapt.a $sym.f.o; \
    mv $sym.f.o ${sym}_netlib.f.o; \
    objcopy --redefine-sym ${sym}_=${sym}_netlib_ ${sym}_netlib.f.o; \
    ar d libscalapack_adapt.a ${sym}.f.o; \
    ar ru libscalapack_adapt.a ${sym}_netlib.f.o; \
    rm ${sym}_netlib.f.o; \
done < comm.sym

(Optional) Generate a single KML_ScaLAPACK dynamic library with complete functions.
In some application scenarios, only a single ScaLAPACK dynamic link library can be linked. In this case, you can use the following method to combine libkscalapack_xxx.a and libscalapack_adapt.a generated previously into a single libkscalapack_full.so.
For example, if the KML_ScaLAPACK library is /HML_LAPACK/build/ins/lib/libkscalapack.a, then the adapted Netlib LAPACK library is libscalapack_adapt.a in the current directory.
```
gcc -o libkscalapack_full.so -shared -fPIC -Wl,--whole-archive /HML_LAPACK/build/ins/lib/libkscalapack.a libscalapack_adapt.a -Wl,--no-whole-archive /HML_LAPACK/build/ins/lib/libklapack.a /HML_LAPACK/build/ins/lib/libkservice.a -fopenmp -lpthread -lgfortran -lm
```
After the execution is complete, the libkscalapack_full.so file is generated in the current directory. You can link the libkscalapack_full.so file to obtain all interface functions of ScaLAPACK v2.2.0.

The KML_BLAS, KML_LAPACK, KML_SERVICE, and system libraries such as libgfortran and libm need to be linked separately.

Verifying the Installation

If the built libraries (libkblas.a and libkblas.so) of KML_BLAS are in KML_BLAS_ROOT, KML_LAPACK is in KML_LAPACK_ROOT. Using KML_BLAS requires the adapted file libscalapack_adapt.a but does not require libblas.a in Netlib ScaLAPACK.

Assume that the environment variable KML_SCALAPACK_ROOT is the directory of libkscalapack.*. Then ADAPT_ROOT is the directory of the adapted Netlib ScaLAPACK library. User applications can use dynamic link libraries or static link libraries.

When compiling the application source code, you need to link to the KML_ScaLAPACK, Netlib ScaLAPACK, KML_LAPACK, Netlib LAPACK, Netlib BLAS, and GFortran libraries and enable OpenMP support.

Option 1: Collaborative KML_BLAS + KML_LAPACK scenario (high performance)

Using the dynamic link library

In this scenario, the Netlib ScaLAPACK has only the static library libscalapack_adapt.a. Therefore, the library is always statically linked.

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /HML_LAPACK/build/ins/lib/ -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -lkblas $KML_LAPACK_ROOT -lklapack -lgfortran -lm -lkservice

The dynamic link libraries required by KML_ScaLAPACK must be found, or /HML_LAPACK/build/ins/lib, $KML_BLAS_ROOT, and $KML_LAPACK_ROOT must be added to LD_LIBRARY_PATH:

if [  ! -n "$LD_LIBRARY_PATH" ]
then
        export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT
else
        export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT:$LD_LIBRARY_PATH
fi

Using the static link library

In this scenario, libblas.a of Netlib LAPACK does not need to be linked.

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /HML_LAPACK/build/ins/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -l:libkblas.a $KML_LAPACK_ROOT -l:klapack.a -l:libkservice.a -l:libgfortran.a -lm

Option 2: Scenario with no KML_BLAS or KML_LAPACK (depending on Netlib LAPACK and BLAS, low performance)

Using the dynamic link library

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /HML_LAPACK/build/ins/lib -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -lblas -llapack -lgfortran -lm -lkservice

The dynamic link libraries required by KML_LAPACK must be found, or /HML_LAPACK/build/ins/lib and $ADAPT_ROOT must be added to LD_LIBRARY_PATH:

if [  ! -n "$LD_LIBRARY_PATH" ]
then
        export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT
else
        export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT:$LD_LIBRARY_PATH
fi

Using the static link library

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /HML_LAPACK/build/ins/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -l:libblas.a -l:liblapack.a -l:libkservice.a -l:libgfortran.a -lm

Parent topic: How to Use KML