Generating a Complete ScaLAPACK Library

Procedure

Download the Netlib ScaLAPACK v2.2.0 source package. Save the package in a path that can be accessed by the compilation device, for example, /data/Download/scalapack-2.2.0.tar.gz.
Download the source package at https://github.com/Reference-ScaLAPACK/scalapack/archive/refs/tags/v2.2.0.tar.gz.

Assume that libkscalapack.a is located in /usr/local/kml/lib. Run the following script to compile the original Netlib ScaLAPACK library. The LAPACK library is required for compiling the Netlib ScaLAPCK library. For details about how to install the LAPACK library, see Generating a Complete LAPACK Library. Assume that the installed complete LAPACK library is located in /openlib/lapack_adapt, and KBLAS is located in /usr/local/kml/lib/kblas/omp. After adding the path of the MPI program to the environment variables, run the following script to generate libscalapack_adapt.a in the scalapack_adapt subdirectory of the current directory:

set -eE
 
echo "SCALAPACK_TGZ          ${SCALAPACK_TGZ:=/data/Download/scalapack-2.2.0.tar.gz}"
echo "LIBKSCALAPACK_A        ${LIBKSCALAPACK_A:=/usr/local/kml/lib/libkscalapack.a}"
echo "LIBKSERVICE_A          ${LIBKSERVICE_A:=/usr/local/kml/lib/libkservice.a}"
echo "ADAPT_DIR              ${ADAPT_DIR:=./scalapack_adapt}"
echo "CMAKE_BUILD_TYPE       ${CMAKE_BUILD_TYPE:=Release}"
echo "LIBSCALAPACK_ADAPT_A   ${LIBSCALAPACK_ADAPT_A:=libscalapack_adapt.a}"
echo "LIBKSCALAPACK_FULL_SO  ${LIBKSCALAPACK_FULL_SO:=libkscalapack_full.so}"
echo "MPICC                  ${MPICC:=mpicc}"
echo "MPIFC                  ${MPIFC:=mpifort}"
echo "LAPACK_SO              ${LAPACK_SO:=/DONT_CARE_OR/openlib/lapack_adapt/libklapack_full.so}"
echo "BLAS_SO                ${BLAS_SO:=/DONT_CARE_OR/usr/local/kml/lib/kblas/omp/libkblas.so}"
 
case $(${MPIFC} -std=legacy - </dev/null 2>&1) in
    *standard*input) export FFLAGS="$FFLAGS -std=legacy" ;;
esac
 
case $(${MPICC} -Wno-implicit-function-declaration - </dev/null 2>&1) in
    *standard*input) export CFLAGS="$CFLAGS -Wno-implicit-function-declaration" ;;
esac
 
mkdir ${ADAPT_DIR}
cd ${ADAPT_DIR}
 
# build netlib scalapack
mkdir netlib
cd netlib
tar xzpf ${SCALAPACK_TGZ}
mkdir build
cd build
cmake_flags=(
    -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON
    -DCMAKE_C_COMPILER=${MPICC}
    -DCMAKE_Fortran_COMPILER=${MPIFC}
    -DLAPACK_LIBRARIES=${LAPACK_SO}
    -DBLAS_LIBRARIES=${BLAS_SO}
    -DLAPACK_FOUND=true
    -DCMAKE_RULE_MESSAGES=off
    -DCMAKE_VERBOSE_MAKEFILE=no
    -DSCALAPACK_BUILD_TESTS=off
)
cmake ${cmake_flags[*]} ../scalapack-2.2.0
make -j
cd ../..
 
cp netlib/build/lib/libscalapack.a ${LIBSCALAPACK_ADAPT_A}
 
# get symbols defined both in kscalapack and netlib scalapack
nm -g ${LIBSCALAPACK_ADAPT_A} | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > netlib.sym
nm -g ${LIBKSCALAPACK_A} | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > kscalapack.sym
comm -12 kscalapack.sym netlib.sym > comm.sym
 
# update symbols name of ${LIBSCALAPACK_ADAPT_A}
while read sym; do
    (
        if ! nm ${LIBSCALAPACK_ADAPT_A} | grep -qe " T ${sym}_\$"; then
            continue
        fi
        ar x ${LIBSCALAPACK_ADAPT_A} ${sym}.f.o
        mv ${sym}.f.o ${sym}_netlib.f.o
        objcopy --redefine-sym ${sym}_=${sym}_netlib_ ${sym}_netlib.f.o
    ) &
done < comm.sym
wait
ar d ${LIBSCALAPACK_ADAPT_A} $(sed -ne 's/$/.f.o/p' comm.sym)
ar ru ${LIBSCALAPACK_ADAPT_A} *_netlib.f.o
rm *_netlib.f.o

(Optional) Generate a standalone KML_ScaLAPACK dynamic library with complete functions.
In some application scenarios, only a single ScaLAPACK dynamic link library can be linked. In this case, you can use the following method to combine libscalapack_adapt.a and libkscalapack.a generated previously into a single libkscalapack_full.so.
For example, if the KML_ScaLAPACK library is /usr/local/kml/lib/libkscalapack.a, then the adapted Netlib ScaLAPACK library is libscalapack_adapt.a in the current directory.
```
${MPIFC} -o ${LIBKSCALAPACK_FULL_SO} -shared -fPIC -Wl,--whole-archive ${LIBKSCALAPACK_A} ${LIBSCALAPACK_ADAPT_A} -Wl,--no-whole-archive ${LIBKSERVICE_A} -fopenmp -lpthread -lm
```
After the execution is complete, the libkscalapack_full.so file is generated in the current directory. You can link the libkscalapack_full.so file to obtain all interface functions of ScaLAPACK v2.2.0.

The KML_BLAS, KML_LAPACK, KML_SERVICE, and system libraries such as libgfortran and libm need to be linked separately.

Verifying the Installation

If the built libraries (libkblas.a and libkblas.so) of KML_BLAS are in KML_BLAS_ROOT, KML_LAPACK is in KML_LAPACK_ROOT. Using KML_BLAS requires the adapted file libscalapack_adapt.a but does not require libblas.a in Netlib ScaLAPACK.

Assume that the environment variable KML_SCALAPACK_ROOT is the directory of libkscalapack.*. Then ADAPT_ROOT is the directory of the adapted Netlib ScaLAPACK library. User applications can use dynamic link libraries or static link libraries.

When compiling the application source code, you need to link to the KML_ScaLAPACK, Netlib ScaLAPACK, KML_LAPACK, Netlib LAPACK, Netlib BLAS, and GFortran libraries and enable OpenMP support.

Option 1: Collaborative KML_BLAS + KML_LAPACK scenario (high performance)

Using the dynamic link library
In this scenario, the Netlib ScaLAPACK has only the static library libscalapack_adapt.a. Therefore, the library is always statically linked.
```
gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/ -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -lkblas $KML_LAPACK_ROOT -lklapack -lgfortran -lm -lkservice
```
The dynamic link libraries required by KML_SCALAPACK must be found, or /usr/local/kml/lib, $KML_BLAS_ROOT, and $KML_LAPACK_ROOT must be added to LD_LIBRARY_PATH.
```
export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT:$LD_LIBRARY_PATH
```

Using the static link library

In this scenario, libblas.a of Netlib LAPACK does not need to be linked.

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -l:libkblas.a $KML_LAPACK_ROOT -l:klapack.a -l:libkservice.a -l:libgfortran.a -lm

Option 2: Scenario with no KML_BLAS or KML_LAPACK (depending on Netlib LAPACK and BLAS, low performance)

Using the dynamic link library

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -lblas -llapack -lgfortran -lm -lkservice

The dynamic link libraries required by KML_LAPACK must be found, or /usr/local/kml/lib and $ADAPT_ROOT must be added to LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT:$LD_LIBRARY_PATH

Using the static link library

gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -l:libblas.a -l:liblapack.a -l:libkservice.a -l:libgfortran.a -lm

Parent topic: How to Use the KML