鲲鹏社区首页
中文
注册
开发者
支持
积分兑换 NEW
我要评分
文档获取效率
文档正确性
内容完整性
文档易理解
提交
在线提单
论坛求助

生成完整的SCALAPACK

KML_SCALAPACK只包含了部分自研SCALAPACK接口，还需要SCALAPACK的其他接口才能运行。因此请自行获取开源的Netlib SCALAPACK v2.2.0的源代码包，并与KML_SCALAPACK适配，以得到全量的SCALAPACK接口。

操作步骤

  1. 下载Netlib ScaLAPACK v2.2.0的源代码包v2.2.0.tar.gz。解压并保存在编译机器可访问的路径中，假设位于“/data/Download/scalapack-2.2.0”，通过以下命令设置环境变量指定Netlib ScaLAPACK源码路径。
    export SCALAPACK_SRC_DIR=/data/Download/scalapack-2.2.0
  2. 假设libkscalapack.so位于“/usr/local/kml/lib/neon”，使用以下脚本编译原始的Netlib ScaLAPACK库。编译Netlib ScaLAPACK库需要LAPACK库，LAPACK库安装请参见生成完整的LAPACK，假设已安装的完整LAPACK库路径为“/path/to/libklapack_full.so”，通过以下命令设置环境变量指定libklapack_full.so的路径：
    export LIBKLAPACK_FULL_SO=/path/to/libklapack_full.so
    KBLAS位于“/usr/local/kml/lib/neon/kblas/multi”，将MPI添加到环境变量后，使用以下脚本可以在当前目录下生成完整功能的libkscalapack_full.so。
    set -eE
# Some helper functions --------------------------------------------------------
function INFORM { printf "\n====== $1\n" ; }
function DIE { printf "\nFATAL ERROR: $*\n" ; exit 1 ; }
INFORM "Define and check build and test parameters ($0)"
echo "LIBKSCALAPACK_SO       ${LIBKSCALAPACK_SO:=/usr/local/kml/lib/neon/libkscalapack.so}"
echo "LIBKBLAS_SO            ${LIBKBLAS_SO:=$(dirname ${LIBKSCALAPACK_SO})/kblas/multi/libkblas.so}"
echo "SCALAPACK_SRC_DIR      ${SCALAPACK_SRC_DIR:=/undefined/SCALAPACK_SRC_DIR}"
echo "BUILD_DIR              ${BUILD_DIR:=${PWD}/tmp-build-scalapack}"
echo "CMAKE_BUILD_TYPE       ${CMAKE_BUILD_TYPE:=Release}"
echo "CTEST_PARALLEL_LEVEL   ${CTEST_PARALLEL_LEVEL:=8}"
echo "OMP_NUM_THREADS        ${OMP_NUM_THREADS:=8}"
echo "MPICC                  ${MPICC:=mpicc}"
echo "MPIFC                  ${MPIFC:=mpifort}"
echo "LIBKLAPACK_FULL_SO     ${LIBKLAPACK_FULL_SO:=${LIBKSCALAPACK_SO/kscalapack.so*/klapack_full.so}}"
echo "LDFLAGS_EXTRA          ${LDFLAGS_EXTRA}"
echo "CMAKEFLAGS_EXTRA       ${CMAKEFLAGS_EXTRA}"
# result
echo "LIBKSCALAPACK_FULL_SO  ${LIBKSCALAPACK_FULL_SO:=./libkscalapack_full.so}"
export CMAKE_BUILD_TYPE CTEST_PARALLEL_LEVEL OMP_NUM_THREADS
test -x ${LIBKSCALAPACK_SO} || DIE "invalid LIBKSCALAPACK_SO"
test -x ${LIBKBLAS_SO} || DIE "invalid LIBKBLAS_SO"
test -r ${SCALAPACK_SRC_DIR}/CMakeLists.txt || DIE "invalid SCALAPACK_SRC_DIR
Download hint:
    git clone https://github.com/Reference-ScaLAPACK/scalapack ${SCALAPACK_SRC_DIR}
"
# Add extra compiler flags -----------------------------------------------------
case $(${MPIFC} -std=legacy - </dev/null 2>&1) in
    *standard*input) export FFLAGS="$FFLAGS -std=legacy" ;;
esac
case $(${MPICC} -Wno-implicit-function-declaration - </dev/null 2>&1) in
    *standard*input) export CFLAGS="$CFLAGS -Wno-implicit-function-declaration" ;;
esac
#-------------------------------------------------------------------------------
INFORM "Configure BUILD_DIR for building STATIC libscalapack.a"
cmake_flags=(
    -DBLAS_LIBRARIES=/do/not/care
    -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
    -DCMAKE_C_COMPILER=${MPICC}
    -DCMAKE_C_FLAGS_RELEASE="-O2 -DNDEBUG="
    -DCMAKE_Fortran_COMPILER=${MPIFC}
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON
    -DCMAKE_RULE_MESSAGES=OFF
    -DCMAKE_VERBOSE_MAKEFILE=no
    -DLAPACK_FOUND=true
    -DLAPACK_LIBRARIES=/do/not/care
    -DSCALAPACK_BUILD_TESTS=off
    ${CMAKEFLAGS_EXTRA}
)
( set -x ; mkdir -p ${BUILD_DIR} )
( set -x ; cd ${BUILD_DIR} ; cmake ${SCALAPACK_SRC_DIR} "${cmake_flags[@]}" )
#-------------------------------------------------------------------------------
INFORM "Build adapted libscalapack.a"
rm ${BUILD_DIR}/lib/libscalapack.a || true
( set -x ; cd ${BUILD_DIR} ; cmake --build . --parallel || cmake --build . --parallel 1 -v )
# ------------------------------------------------------------------------------
INFORM "Get symbols defined BOTH in KSCALAPACK and in Reference ScaLAPACK"
nm -g ${BUILD_DIR}/lib/libscalapack.a | grep -oP ' T \K\w+(?=_$)' | sort | uniq > netlib.sym
nm -g ${LIBKSCALAPACK_SO/kscalapack.so*/kscalapack.a} | grep -oP ' T \K\w+(?=_$)' | sort | uniq > kscalapack.sym
comm -12 kscalapack.sym netlib.sym > comm.sym
# ------------------------------------------------------------------------------
INFORM "Extract relevant modules and update symbol names"
( set -x ; mkdir -p ${BUILD_DIR}/mods )
while read sym; do
    (
        cd ${BUILD_DIR}/mods
        tloc=$(nm -gA --defined-only ${BUILD_DIR}/lib/libscalapack.a | grep " T ${sym}_\$")
        if [[ -n "${tloc}" ]]; then
            libmod=${tloc%:*}
            mod=${libmod#*:}
            ( set -x ; ar x ${BUILD_DIR}/lib/libscalapack.a ${mod} )
            ( set -x ; objcopy --redefine-sym ${sym}_=${sym}_netlib_ ${mod} )
        fi
    ) &
done < comm.sym
wait
ar d ${BUILD_DIR}/lib/libscalapack.a $(cd ${BUILD_DIR}/mods ; ls *.o)
( cd ${BUILD_DIR}/mods ; ar ru ${BUILD_DIR}/lib/libscalapack.a *.o )
( cd ${BUILD_DIR}/mods ; rm *.o )
#-------------------------------------------------------------------------------
INFORM "Create libkscalapack_full.so"
( set -x ; mkdir -p $(dirname ${LIBKSCALAPACK_FULL_SO}))
ldflags=(
    -Wl,--whole-archive
    -Wl,--start-group
    ${LIBKSCALAPACK_SO/kscalapack.so*/kscalapack.a}
    ${BUILD_DIR}/lib/libscalapack.a
    ${LIBKSCALAPACK_SO/kscalapack.so*/kservice.a}
    -Wl,--end-group
    -Wl,--no-whole-archive
    ${LDFLAGS_EXTRA}
)
kmlver=$(strings ${LIBKSCALAPACK_SO/kscalapack.so*/kservice.so} | grep -o 'VERSION.*COMPILER.*')
INFORM "Check if compilers are compatible with KML '${kmlver}'"
function check_compiler {
    local name=$1 parlor=$2
    case `${!name} --version 2>/dev/null` in
    *${parlor}*) echo "  Using ${name}=${!name}" ;;
    '') DIE "Missing compiler ${name}=${!name}" ;;
    *) DIE "Wrong compiler ${name}=${!name}" ;;
    esac
}
case ${kmlver} in
*COMPILER:GNU*)
    export CC=${CC:-gcc} FC=${FC:-gfortran}
    check_compiler CC 'gcc'
    check_compiler FC 'GNU Fortran'
    ( set -x ; ${MPIFC} "${ldflags[@]}" -lstdc++ -shared -o ${LIBKSCALAPACK_FULL_SO} )
    ;;
*COMPILER:Clang*)
    export CC=${CC:-clang} FC=${FC:-flang}
    check_compiler CC 'clang'
    check_compiler FC '?lang'
    ( set -x ; ${MPIFC} "${ldflags[@]}" -lc++ -shared -o ${LIBKSCALAPACK_FULL_SO} )
    ;;
*)
    DIE "Unknown KML version ${kmlver}"
    ;;
esac
#-------------------------------------------------------------------------------
if [[ "$1" == "test" ]]; then
    INFORM "Fixup CMakeLists in SCALAPACK_SRC_DIR/TESTING"
    ( set -x ; sed -i -e 's/ scalapack / ${SCALAPACK_LIBRARIES} /' ${SCALAPACK_SRC_DIR}/TESTING/*/CMakeLists.txt )
    INFORM "Run tests with libkscalapack_full.so"
    function tests {
        local tmp_dir=$1 scalapack_libs=$2
        cmake_flags=(
            -DBUILD_TESTING=ON
            -DBLAS_LIBRARIES="${LIBKBLAS_SO}"
            -DLAPACK_LIBRARIES="${LIBKLAPACK_FULL_SO}"
            -DSCALAPACK_LIBRARIES="${scalapack_libs}"
            -DCMAKE_RULE_MESSAGES=OFF
            -DSCALAPACK_BUILD_TESTS=ON
            -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
            -DCMAKE_C_COMPILER=${MPICC}
            -DCMAKE_C_FLAGS_RELEASE="-O2 -DNDEBUG="
            -DCMAKE_Fortran_COMPILER=${MPIFC}
        )
        mkdir -p ${tmp_dir}
        ( set -x ; cd ${tmp_dir} ; cmake ${SCALAPACK_SRC_DIR} "${cmake_flags[@]}" )
        ( set -x ; cd ${tmp_dir} ; cmake --build . --parallel || cmake --build . --parallel 1 -v )
        ( set -x ; cd ${tmp_dir} ; ctest --output-on-failure -V)
        echo "FINISHED SCALAPACK TESTS IN ${tmp_dir}"
    }
    tests ${BUILD_DIR}-tests "${LIBKSCALAPACK_FULL_SO};${LIBKBLAS_SO};-fopenmp"
fi
# Report build results ---------------------------------------------------------
cat <<EOF
--------------------------------------------------------------------------------
FOR THIS KSCALAPACK LIBRARY (${kmlver}):
$(ls -lU ${LIBKSCALAPACK_SO} ${LIBKSCALAPACK_SO/kscalapack.so*/kscalapack.a})
WE HAVE BUILT FULL SCALAPACK LIBRARY:
$(ls -lU ${LIBKSCALAPACK_FULL_SO})
--------------------------------------------------------------------------------
EOF
  3. 执行完成后当前目录下会生成libkscalapack_full.so，可以单独链接此so得到SCALAPACK v2.2.0的全部接口功能。

    此时仍然需要单独链接KML_BLAS库、LAPACK_FULL库、KML_SERVICE库。

安装后验证

假设KML_BLAS已构建的库（libkblas.a和libkblas.so）位于KML_BLAS_ROOT中，KML_LAPACK位于KML_LAPACK_ROOT中。当使用KML_BLAS时，仍然需要上一节最后获得Netlib SCALAPACK适配的libscalapack_adapt.a，但不再需要Netlib SCALAPACK中的libblas.a。

使用时，假设环境变量KML_SCALAPACK_ROOT为libkscalapack.*所在目录，ADAPT_ROOT为适配后Netlib SCALAPACK库所在目录。用户应用可以选择使用动态链接库或静态链接库。

编译应用源代码时需要链接到KML_SCALAPACK、Netlib SCALAPACK、KML_LAPACK、Netlib LAPACK、Netlib BLAS和gfortran库并开启OpenMP支持。

选项一、协同KML_BLAS和KML_LAPACK场景（高性能）

  • 使用动态链接库

    此场景下由于适配的Netlib SCALAPACK只有静态库libscalapack_adapt.a，因此总是静态链接到此库。

    gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/ -L /usr/local/kml/lib/neon -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -lkblas $KML_LAPACK_ROOT -lklapack -lgfortran -lm -lkservice

    运行时需能找到KML_SCALAPACK所需的这些动态链接库，或者将/usr/local/kml/lib和$KML_BLAS_ROOT、$KML_LAPACK_ROOT加入LD_LIBRARY_PATH。

    export LD_LIBRARY_PATH=/usr/local/kml/lib:$KML_BLAS_ROOT:$KML_LAPACK_ROOT:$LD_LIBRARY_PATH
  • 使用静态链接库

    此场景不再需要链接Netlib LAPACK的libblas.a。

    gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/ -L /usr/local/kml/lib/neon -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -L $KML_BLAS_ROOT -l:libkblas.a $KML_LAPACK_ROOT -l:klapack.a -l:libkservice.a -l:libgfortran.a -lm

    其中app.c为示例代码，请参见后续章节“KML_SCALAPACK库函数说明>矩阵分解函数”中对应函数的代码示例。

选项二、无KML_BLAS和KML_LAPACK场景（依赖Netlib LAPACK以及自带的BLAS，性能较低）

  • 使用动态链接库
    gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/ -L /usr/local/kml/lib/neon -lkscalapack -L $ADAPT_ROOT -l:libscalapack_adapt.a -lblas -llapack -lgfortran -lm -lkservice

    运行时需能找到KML_LAPACK所需的这些动态链接库，或者将/usr/local/kml/lib和$ADAPT_ROOT加入LD_LIBRARY_PATH。

    export LD_LIBRARY_PATH=/usr/local/kml/lib:$ADAPT_ROOT:$LD_LIBRARY_PATH
  • 使用静态链接库
    gcc app.c -o app -fopenmp -I $KML_LAPACK_ROOT/include/ -L /usr/local/kml/lib/neon -l:libkscalapack.a -L $ADAPT_ROOT -l:libscalapack_adapt.a -l:libblas.a -l:liblapack.a -l:libkservice.a -l:libgfortran.a -lm
父主题： 如何使用KML