生成完整的SCALAPACK
KML_SCALAPACK只包含了部分自研SCALAPACK接口,还需要SCALAPACK的其他接口才能运行。因此请自行获取开源的Netlib SCALAPACK v2.2.0的源代码包,并与KML_SCALAPACK适配,以得到全量的SCALAPACK接口。
操作步骤
- 下载Netlib SCALAPACK v2.2.0的源代码包。保存在编译机器可访问的路径中,假设位于“/data/Download/scalapack-2.2.0.tar.gz”。
源代码包获取路径:https://github.com/Reference-ScaLAPACK/scalapack/archive/refs/tags/v2.2.0.tar.gz
- 假设libkscalapack.a位于“/usr/local/kml/lib”,使用以下脚本编译原始的Netlib ScaLAPACK库。编译Netlib ScaLAPCK库需要LAPACK库,LAPACK库安装请参见生成完整的LAPACK,假设已安装的完整LAPACK库位于“/openlib/lapack_adapt”,KBLAS位于“/usr/local/kml/lib/kblas/omp”,将MPI添加到环境变量后,使用以下脚本可以在当前目录的scalapack_adapt子目录下生成适配的libscalapack_adapt.a。
set -eE echo "SCALAPACK_TGZ ${SCALAPACK_TGZ:=/data/Download/scalapack-2.2.0.tar.gz}" echo "LIBKSCALAPACK_A ${LIBKSCALAPACK_A:=/usr/local/kml/lib/libkscalapack.a}" echo "LIBKSERVICE_A ${LIBKSERVICE_A:=/usr/local/kml/lib/libkservice.a}" echo "ADAPT_DIR ${ADAPT_DIR:=./scalapack_adapt}" echo "CMAKE_BUILD_TYPE ${CMAKE_BUILD_TYPE:=Release}" echo "LIBSCALAPACK_ADAPT_A ${LIBSCALAPACK_ADAPT_A:=libscalapack_adapt.a}" echo "LIBKSCALAPACK_FULL_SO ${LIBKSCALAPACK_FULL_SO:=libkscalapack_full.so}" echo "MPICC ${MPICC:=mpicc}" echo "MPIFC ${MPIFC:=mpifort}" echo "LAPACK_SO ${LAPACK_SO:=/DONT_CARE_OR/openlib/lapack_adapt/libklapack_full.so}" echo "BLAS_SO ${BLAS_SO:=/DONT_CARE_OR/usr/local/kml/lib/kblas/omp/libkblas.so}" case $(${MPIFC} -std=legacy - </dev/null 2>&1) in *standard*input) export FFLAGS="$FFLAGS -std=legacy" ;; esac case $(${MPICC} -Wno-implicit-function-declaration - </dev/null 2>&1) in *standard*input) export CFLAGS="$CFLAGS -Wno-implicit-function-declaration" ;; esac mkdir ${ADAPT_DIR} cd ${ADAPT_DIR} # build netlib scalapack mkdir netlib cd netlib tar xzpf ${SCALAPACK_TGZ} mkdir build cd build cmake_flags=( -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER=${MPICC} -DCMAKE_Fortran_COMPILER=${MPIFC} -DLAPACK_LIBRARIES=${LAPACK_SO} -DBLAS_LIBRARIES=${BLAS_SO} -DLAPACK_FOUND=true -DCMAKE_RULE_MESSAGES=off -DCMAKE_VERBOSE_MAKEFILE=no -DSCALAPACK_BUILD_TESTS=off ) cmake ${cmake_flags[*]} ../scalapack-2.2.0 make -j cd ../.. cp netlib/build/lib/libscalapack.a ${LIBSCALAPACK_ADAPT_A} # get symbols defined both in kscalapack and netlib scalapack nm -g ${LIBSCALAPACK_ADAPT_A} | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > netlib.sym nm -g ${LIBKSCALAPACK_A} | grep 'T ' | grep -oP '\K\w+(?=_$)' | sort | uniq > kscalapack.sym comm -12 kscalapack.sym netlib.sym > comm.sym # update symbols name of ${LIBSCALAPACK_ADAPT_A} while read sym; do ( if ! nm ${LIBSCALAPACK_ADAPT_A} | grep -qe " T ${sym}_\$"; then continue fi ar x ${LIBSCALAPACK_ADAPT_A} ${sym}.f.o mv ${sym}.f.o ${sym}_netlib.f.o objcopy --redefine-sym ${sym}_=${sym}_netlib_ ${sym}_netlib.f.o ) & done < comm.sym wait ar d ${LIBSCALAPACK_ADAPT_A} $(sed -ne 's/$/.f.o/p' comm.sym) ar ru ${LIBSCALAPACK_ADAPT_A} *_netlib.f.o rm *_netlib.f.o
- (可选)生成完整功能的单一KML_SCALAPACK动态库。
某些应用场景可能只允许链接单个SCALAPACK动态链接库,此时可采用以下方法将上一节生成的libscalapack_adapt.a和libkscalapack.a合成单一的libkscalapack_full.so。
假设KML_SCALAPACK库为“/usr/local/kml/lib/libkscalapack.a”,适配后的Netlib ScaLAPACK库为当前目录下的libscalapack_adapt.a。${MPIFC} -o ${LIBKSCALAPACK_FULL_SO} -shared -fPIC -Wl,--whole-archive ${LIBKSCALAPACK_A} ${LIBSCALAPACK_ADAPT_A} -Wl,--no-whole-archive ${LIBKSERVICE_A} -fopenmp -lpthread -lm
执行完成后当前目录下会生成libkscalapack_full.so,可以单独链接此so得到SCALAPACK v2.2.0的全部接口功能。此时仍然需要单独链接KML_BLAS库、KML_LAPACK库、KML_SERVICE库以及libgfortran、libm等系统库。
安装后验证
假设KML_BLAS已构建的库(libkblas.a和libkblas.so)位于KML_BLAS_ROOT中,KML_LAPACK位于KML_LAPACK_ROOT中。当使用KML_BLAS时,仍然需要上一节最后获得Netlib SCALAPACK适配的libscalapack_adapt.a,但不再需要Netlib SCALAPACK中的libblas.a。
使用时,假设环境变量KML_SCALAPACK_ROOT为libkscalapack.*所在目录,ADAPT_ROOT为适配后Netlib SCALAPACK库所在目录。用户应用可以选择使用动态链接库或静态链接库。
编译应用源代码时需要链接到KML_SCALAPACK、Netlib SCALAPACK、KML_LAPACK、Netlib LAPACK、Netlib BLAS和gfortran库并开启OpenMP支持。
选项一、协同KML_BLAS和KML_LAPACK场景(高性能)
- 使用动态链接库
此场景下由于适配的Netlib SCALAPACK只有静态库libscalapack_adapt.a,因此总是静态链接到此库。
gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/ -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -lkblas $KML_LAPACK_ROOT -lklapack -lgfortran -lm -lkservice
运行时需能找到KML_SCALAPACK所需的这些动态链接库,或者将/usr/local/kml/lib和$KML_BLAS_ROOT、$KML_LAPACK_ROOT加入LD_LIBRARY_PATH。
export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT:$LD_LIBRARY_PATH
- 使用静态链接库
此场景不再需要链接Netlib LAPACK的libblas.a。
gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -l:libkblas.a $KML_LAPACK_ROOT -l:klapack.a -l:libkservice.a -l:libgfortran.a -lm
选项二、无KML_BLAS和KML_LAPACK场景(依赖Netlib LAPACK以及自带的BLAS,性能较低)
- 使用动态链接库
gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -lblas -llapack -lgfortran -lm -lkservice
运行时需能找到KML_LAPACK所需的这些动态链接库,或者将/usr/local/kml/lib和$ADAPT_ROOT加入LD_LIBRARY_PATH。
export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT:$LD_LIBRARY_PATH
- 使用静态链接库
gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/kml-0.3.0 -L /usr/local/kml/lib -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -l:libblas.a -l:liblapack.a -l:libkservice.a -l:libgfortran.a -lm