Generating a Complete ScaLAPACK Library

Procedure

1. Download the Netlib ScaLAPACK v2.2.0 source package. Save the file in a path that can be accessed by the compilation device, for example, /openlib/scalapack-2.2.0.tar.gz.

   Download the source package at https://github.com/Reference-ScaLAPACK/scalapack/archive/refs/tags/v2.2.0.tar.gz.

2. Assume that libkscalapack.a is in /usr/local/kml/lib/. Run the following script to compile the original Netlib ScaLAPACK library. To compile Netlib ScaLAPACK, you need to install the Netlib LAPACK library first (refer to Generating a Complete LAPACK Library). For example, if the Netlib LAPACK library is in /openlib/lapack-3.11/, then libscalapack_adapt.a will be generated in the scalapack_adapt subdirectory of the current directory.

   netlib=/openlib/scalapack-2.2.0.tar.gz
   kscalapack=/usr/local/kml/lib/libkscalapack.a
   libLapackPath=/openlib/lapack_adapt/libklapack_full.so
   libRefBlasPath=/usr/local/kml/lib/kblas/omp/libkblas.so
   
   mkdir scalapack_adapt
   cd scalapack_adapt
   
   # build netlib scalapack
   mkdir netlib
   cd netlib
   tar zxvf $netlib
   mkdir build
   cd build
   cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort \
   -DLAPACK_LIBRARIES=/$libLapackPath -DBLAS_LIBRARIES=/$libRefBlasPath ../scalapack-2.2.0
   make -j
   cd ../..
   
   cp netlib/build/lib/libscalapack.a libscalapack_adapt.a
   
   # get symbols defined both in kscalapack and netlib scalapack
   nm -g libscalapack_adapt.a | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > netlib.sym
   nm -g $kscalapack | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > kscalapack.sym
   comm -12 kscalapack.sym netlib.sym > comm.sym 
   
   # update symbols name of libscalapack_adapt.a
   while read sym; do \
       if ! nm libscalapack_adapt.a | grep -qe " T ${sym}_\$"; then \
           continue; \
       fi; \
       ar x libscalapack_adapt.a $sym.f.o; \
       mv $sym.f.o ${sym}_netlib.f.o; \
       objcopy --redefine-sym ${sym}_=${sym}_netlib_ ${sym}_netlib.f.o; \
       ar d libscalapack_adapt.a ${sym}.f.o; \
       ar ru libscalapack_adapt.a ${sym}_netlib.f.o; \
       rm ${sym}_netlib.f.o; \
   done < comm.sym

3. (Optional) Generate a standalone KML_ScaLAPACK dynamic library with complete functions.

   In some application scenarios, only a single ScaLAPACK dynamic link library can be linked. In this case, you can use the following method to combine libkscalapack_xxx.a and libscalapack_adapt.a generated previously into a single libkscalapack_full.so.

   For example, if the KML_ScaLAPACK library is /usr/local/kml/lib/libkscalapack.a, then the adapted Netlib ScaLAPACK library is libscalapack_adapt.a in the current directory.

   gcc -o libkscalapack_full.so -shared -fPIC -Wl,--whole-archive /usr/local/kml/lib/libkscalapack.a libscalapack_adapt.a -Wl,--no-whole-archive /usr/local/kml/lib/libkservice.a -fopenmp -lpthread -lgfortran -lm

   After the execution is complete, the libkscalapack_full.so file is generated in the current directory. You can link the libkscalapack_full.so file to obtain all interface functions of ScaLAPACK v2.2.0.

   

   The KML_BLAS, KML_LAPACK, KML_SERVICE, and system libraries such as libgfortran and libm need to be linked separately.

Verifying the Installation

If the built libraries (libkblas.a and libkblas.so) of KML_BLAS are in KML_BLAS_ROOT, KML_LAPACK is in KML_LAPACK_ROOT. Using KML_BLAS requires the adapted file libscalapack_adapt.a but does not require libblas.a in Netlib ScaLAPACK.

Assume that the environment variable KML_SCALAPACK_ROOT is the directory of libkscalapack.*. Then ADAPT_ROOT is the directory of the adapted Netlib ScaLAPACK library. User applications can use dynamic link libraries or static link libraries.

When compiling the application source code, you need to link to the KML_ScaLAPACK, Netlib ScaLAPACK, KML_LAPACK, Netlib LAPACK, Netlib BLAS, and GFortran libraries and enable OpenMP support.

Option 1: Collaborative KML_BLAS + KML_LAPACK scenario (high performance)

• Using the dynamic link library

  In this scenario, the Netlib ScaLAPACK has only the static library libscalapack_adapt.a. Therefore, the library is always statically linked.

  gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/ -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -lkblas $KML_LAPACK_ROOT -lklapack -lgfortran -lm -lkservice

  The dynamic link libraries required by KML_SCALAPACK must be found, or /usr/local/kml/lib, $KML_BLAS_ROOT, and $KML_LAPACK_ROOT must be added to LD_LIBRARY_PATH.

  export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT:$LD_LIBRARY_PATH

• Using the static link library

  In this scenario, libblas.a of Netlib LAPACK does not need to be linked.

  gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -l:libkblas.a $KML_LAPACK_ROOT -l:klapack.a -l:libkservice.a -l:libgfortran.a -lm

Option 2: Scenario with no KML_BLAS or KML_LAPACK (depending on Netlib LAPACK and BLAS, low performance)

• Using the dynamic link library

  gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -lblas -llapack -lgfortran -lm -lkservice

  The dynamic link libraries required by KML_LAPACK must be found, or /usr/local/kml/lib and $ADAPT_ROOT must be added to LD_LIBRARY_PATH.

  export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT:$LD_LIBRARY_PATH

• Using the static link library

  gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -l:libblas.a -l:liblapack.a -l:libkservice.a -l:libgfortran.a -lm