Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing Hyper MPI Using Source Code

Prerequisites

The following dependencies have been installed:
  • perl-Data-Dumper
  • autoconf
  • automake
  • libtool 2.4.2
  • GCC 9.3.0/BiSheng Compiler 2.3.0

    Before installing GCC 9.3.0/BiSheng Compiler 2.3.0, install the following dependencies in sequence: gmp-6.2.0.tar.bz2, mpfr-4.0.2.tar.bz2, mpc-1.1.0.tar.gz, and isl-0.18.tar.bz2.

  • numactl
  • binutils
  • systemd-devel
  • valgrind
  • cmake3

Checking Compilation Environment Variables

  1. Use PuTTY to log in to the job execution node as the Hyper MPI installation and maintenance user, for example, hmpi_master.
  2. Check whether GCC 9.3.0/BiSheng Compiler 2.3.0 has been installed.
    • GCC 9.3.0

      gcc -v

      Check whether GCC 9.3.0 is displayed in the command output:

    • BiSheng Compiler 2.3.0

      clang -v

      Check whether the command output contains the BiSheng Compiler information and whether the compiler version is correct.

  3. Run the following command to check whether libtool has been installed:

    libtool --version

    Check whether libtool 2.4.2 is displayed in the command output:

Compiling the Hyper MPI Software Package Using GCC

Hyper MPI consists of two software layers: Hyper MPI and HUCX. The Hyper MPI installation depends on HUCX. Build HUCX before building Hyper MPI.

  • Building HUCX
    1. Use PuTTY to log in to the job execution node as the Hyper MPI installation and maintenance user, for example, hmpi_master.
    2. Download the HUCX source package ucx-1.10.1.tar.gz.

      https://github.com/openucx/ucx/archive/refs/tags/v1.10.1.tar.gz

    3. Upload the source package to the installation directory of the HUCX source package on the server, for example, /path/to/download/hucx.
    4. Run the following command to go to the installation directory of the HUCX source package:

      cd /path/to/download/hucx

    5. Run the following command to decompress the source package:

      tar -zxvf ucx-1.10.1.tar.gz

    6. Run the following command to go to the directory where the autogen.sh script is stored:

      cd ucx-1.10.1/

    7. Run the following commands to install the source package:

      ./autogen.sh

      ./contrib/configure-opt --prefix=/path/to/install/hucx --enable-mt CC=gcc CXX=g++ FC=gfortran

      • If GPU-aware is required and CUDA has been installed in the environment, run the following command to specify the --with-cuda option:

        ./contrib/configure-opt --prefix=/path/to/install/hucx --with-cuda=/path/to/install/cuda --enable-mt CC=gcc CXX=g++ FC=gfortran

      • If the numactl plugin has not been installed, run the following command to specify the --disable-numa parameter:

        ./contrib/configure-opt --prefix=/path/to/install/hucx --disable-numa --enable-mt CC=gcc CXX=g++ FC=gfortran

      • /path/to/install/hucx indicates the HUCX installation path.
      • /path/to/install/cuda indicates the CUDA installation path.

      make -j32

      make -j32 install

      The -j parameter following make is a parallel compilation parameter, indicating the number of CPU cores involved in the compilation process. Specify this parameter based on the actual number of CPU cores.

    8. Run the following command to load the HUCX library:

      export LD_LIBRARY_PATH=/path/to/install/hucx/lib:$LD_LIBRARY_PATH

      export C_INCLUDE_PATH=/path/to/install/hucx/include:$C_INCLUDE_PATH

      export CPLUS_INCLUDE_PATH=/path/to/install/hucx/include:$CPLUS_INCLUDE_PATH

  • Building XUCG
    1. Download the XUCG source package xucg-v1.3.0-huawei.tar.gz.

      https://gitee.com/kunpengcompute/xucg/archive/refs/tags/v1.3.0-huawei.tar.gz

    2. Upload the source package to the installation directory of the XUCG source package on the server, for example, /path/to/download/xucg.
    3. Run the following command to go to the installation directory of the XUCG source package:

      cd /path/to/download/xucg

    4. Run the following command to decompress the source package:

      tar -zxvf xucg-v1.3.0-huawei.tar.gz

    5. Run the following command to go to the directory where CMakeLists.txt is located:

      cd xucg-v1.3.0-huawei/

    6. Run the following commands to install the source package:

      mkdir build && cd build

      cmake3 .. -DCMAKE_INSTALL_PREFIX=/path/to/install/xucg -DCMAKE_BUILD_TYPE=Release -DUCG_BUILD_WITH_UCX=/path/to/install/hucx -DUCG_ENABLE_MT=ON -DUCG_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran

      /path/to/install/xucg indicates the XUCG installation path.

      make -j32

      make -j32 install

      The -j parameter following make is a parallel compilation parameter, indicating the number of CPU cores involved in the compilation process. Specify this parameter based on the actual number of CPU cores.

    7. Run the following command to load the XUCG library:

      export LD_LIBRARY_PATH=/path/to/install/xucg/lib:$LD_LIBRARY_PATH

      export C_INCLUDE_PATH=/path/to/install/xucg/include:$C_INCLUDE_PATH

      export CPLUS_INCLUDE_PATH=/path/to/install/xucg/include:$CPLUS_INCLUDE_PATH

  • Building Hyper MPI
    1. Use PuTTY to log in to the job execution node as the Hyper MPI installation and maintenance user, for example, hmpi_master.
    2. Download the Hyper MPI source package hmpi-v1.3.0-huawei.tar.gz.

      https://gitee.com/kunpengcompute/hmpi/archive/refs/tags/v1.3.0-huawei.tar.gz

    3. Upload the source package to its installation directory on the server, for example, /path/to/download/hmpi.
    4. Run the following command to go to the installation directory:

      cd /path/to/download/hmpi

    5. Run the following command to decompress the software package:

      tar -zxvf hmpi-v1.3.0-huawei.tar.gz

    6. Run the following command to go to the directory where the autogen.pl script is stored:

      cd hmpi-v1.3.0-huawei/

    7. Run the following commands to install the Hyper MPI source package:

      ./autogen.pl

      ./configure --prefix=/path/to/install/hmpi --with-platform=contrib/platform/mellanox/optimized --enable-mpi1-compatibility --with-ucx=/path/to/install/hucx --with-ucg=/path/to/install/xucg CC=gcc CXX=g++ FC=gfortran

      make

      make install

      • If GPU-aware is required and CUDA has been installed in the environment, run the following command to specify the --with-cuda option:

        ./configure --prefix=/path/to/install/hmpi --with-platform=contrib/platform/mellanox/optimized --with-cuda=/path/to/install/cuda --enable-mpi1-compatibility --with-ucx=/path/to/install/hucx --with-ucg=/path/to/install/xucg CC=gcc CXX=g++ FC=gfortran

      • /path/to/install/hmpi indicates the Hyper MPI installation path.
      • /path/to/install/cuda indicates the CUDA installation path.
      • /path/to/install/hucx indicates the HUCX installation path.
      • /path/to/install/xucg indicates the XUCG installation path.

Compiling the Hyper MPI Software Package Using BiSheng Compiler

Hyper MPI consists of two software layers: Hyper MPI and HUCX. The Hyper MPI installation depends on HUCX. Build HUCX before building Hyper MPI.

  • Building HUCX
    1. Use PuTTY to log in to the job execution node as the Hyper MPI installation and maintenance user, for example, hmpi_master.
    2. Download the HUCX source package ucx-1.10.1.tar.gz.

      https://github.com/openucx/ucx/archive/refs/tags/v1.10.1.tar.gz

    3. Upload the source package to the installation directory of the HUCX source package on the server, for example, /path/to/download/hucx.
    4. Run the following command to go to the installation directory of the HUCX source package:

      cd /path/to/download/hucx

    5. Run the following command to decompress the source package:

      tar -zxvf ucx-1.10.1.tar.gz

    6. Run the following command to go to the directory where the autogen.sh script is stored:

      cd ucx-1.10.1/

    7. Run the following commands to install the source package:

      ./autogen.sh

      ./contrib/configure-opt --prefix=/path/to/install/hucx --enable-mt CC=clang CXX=clang++ FC=flang

      • If GPU-aware is required and CUDA has been installed in the environment, run the following command to specify the --with-cuda option:

        ./contrib/configure-opt --prefix=/path/to/install/hucx --with-cuda=/path/to/install/cuda --enable-mt CC=clang CXX=clang++ FC=flang

      • If the numactl plugin has not been installed, run the following command to specify the --disable-numa parameter:

        ./contrib/configure-opt --prefix=/path/to/install/hucx --disable-numa --enable-mt CC=clang CXX=clang++ FC=flang

      • /path/to/install/hucx indicates the HUCX installation path.
      • /path/to/install/cuda indicates the CUDA installation path.

      make -j32

      make -j32 install

      The -j parameter following make is a parallel compilation parameter, indicating the number of CPU cores involved in the compilation process. Specify this parameter based on the actual number of CPU cores.

    8. Run the following command to load the HUCX library:

      export LD_LIBRARY_PATH=/path/to/install/hucx/lib:$LD_LIBRARY_PATH

      export C_INCLUDE_PATH=/path/to/install/hucx/include:$C_INCLUDE_PATH

      export CPLUS_INCLUDE_PATH=/path/to/install/hucx/include:$CPLUS_INCLUDE_PATH

  • Building XUCG
    1. Download the XUCG source package xucg-v1.3.0-huawei.tar.gz.

      https://gitee.com/kunpengcompute/xucg/archive/refs/tags/v1.3.0-huawei.tar.gz

    2. Upload the source package to the installation directory of the XUCG source package on the server, for example, /path/to/download/xucg.
    3. Run the following command to go to the installation directory of the XUCG source package:

      cd /path/to/download/xucg

    4. Run the following command to decompress the source package:

      tar -zxvf xucg-v1.3.0-huawei.tar.gz

    5. Run the following command to go to the directory where CMakeLists.txt is located:

      cd xucg-v1.3.0-huawei/

    6. Run the following commands to install the source package:

      mkdir build && cd build

      cmake3 .. -DCMAKE_INSTALL_PREFIX=/path/to/install/xucg -DCMAKE_BUILD_TYPE=Release -DUCG_BUILD_WITH_UCX=/path/to/install/hucx -DUCG_ENABLE_MT=ON -DUCG_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_Fortran_COMPILER=flang

      /path/to/install/xucg indicates the XUCG installation path.

      make -j32

      make -j32 install

      The -j parameter following make is a parallel compilation parameter, indicating the number of CPU cores involved in the compilation process. Specify this parameter based on the actual number of CPU cores.

    7. Run the following command to load the XUCG library:

      export LD_LIBRARY_PATH=/path/to/install/xucg/lib:$LD_LIBRARY_PATH

      export C_INCLUDE_PATH=/path/to/install/xucg/include:$C_INCLUDE_PATH

      export CPLUS_INCLUDE_PATH=/path/to/install/xucg/include:$CPLUS_INCLUDE_PATH

    Building Hyper MPI

    1. Use PuTTY to log in to the job execution node as the Hyper MPI installation and maintenance user, for example, hmpi_master.
    2. Download the Hyper MPI source package hmpi-v1.3.0-huawei.tar.gz.

      https://gitee.com/kunpengcompute/hmpi/archive/refs/tags/v1.3.0-huawei.tar.gz

    3. Upload the source package to its installation directory on the server, for example, /path/to/download/hmpi.
    4. Run the following command to go to the installation directory:

      cd /path/to/download/hmpi

    5. Run the following command to decompress the software package:

      tar -zxvf hmpi-v1.3.0-huawei.tar.gz

    6. Run the following command to go to the directory where the autogen.pl script is stored:

      cd hmpi-v1.3.0-huawei/

    7. Run the following commands to install the Hyper MPI source package:

      ./autogen.pl

      ./configure --prefix=/path/to/install/hmpi --with-platform=contrib/platform/mellanox/optimized --enable-mpi1-compatibility --with-ucx=/path/to/install/hucx --with-ucg=/path/to/install/xucg CC=clang CXX=clang++ FC=flang

      make

      make install

      • If GPU-aware is required and CUDA has been installed in the environment, run the following command to specify the --with-cuda option:

        ./configure --prefix=/path/to/install/hmpi --with-platform=contrib/platform/mellanox/optimized --with-cuda=/path/to/install/cuda --enable-mpi1-compatibility --with-ucx=/path/to/install/hucx --with-ucg=/path/to/install/xucg CC=clang CXX=clang++ FC=flang

      • /path/to/install/hmpi indicates the Hyper MPI installation path.
      • /path/to/install/cuda indicates the CUDA installation path.
      • /path/to/install/hucx indicates the HUCX installation path.
      • /path/to/install/xucg indicates the XUCG installation path.

Configuring Environment Variables

To use MPI, you need to configure environment variables. You can use the environment-modules tool to configure environment variables.
  1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
  2. Run the following command to install the environment-modules tool:

    yum install -y environment-modules

  3. Create and edit the Hmpi_modulefiles configuration file in the environment-modules installation directory:
    1. Open the Hmpi_modulefiles configuration file.

      vi /path/to/install/Hmpi_modulefiles

    2. Press i to enter the insert mode and modify the following content:
      #%Module1.0
#  HMPI module for use with 'environment-modules' package:
conflict mpi 
prepend-path  OPAL_PREFIX /path/to/install/hmpi/
prepend-path  PATH /path/to/install/hmpi/bin:/path/to/install/hucx/bin:/path/to/install/xucg/bin
prepend-path  LD_LIBRARY_PATH /path/to/install/hmpi/lib:/path/to/install/hucx/lib:/path/to/install/xucg/lib
prepend-path  INCLUDE /path/to/install/hmpi/include:/path/to/install/hucx/include:/path/to/install/xucg/include

      /path/to/install indicates the installation path of the environment-modules tool. If no installation path is specified, the default installation path /usr/share/Modules/modulefiles is used.

    3. Press Esc, type :wq!, and press Enter to save the settings and exit.
  4. Run the following module commands to load environment variables in the tool installation path:

    module use /path/to/install

    module load /path/to/install/Hmpi_modulefiles

    If the module command is unavailable, run the /usr/share/Modules/init/bash command for the command to take effect.

  5. Write the environment variables of Hyper MPI to the ~/.bashrc file.
    1. Open the ~/.bashrc file.

      vi ~/.bashrc

    2. Press i to enter the insert mode and add the following content:
      module use /path/to/install
module load /path/to/install/Hmpi_modulefiles
    3. Press Esc, type :wq!, and press Enter to save the settings and exit.

      The ~/.bashrc file of the default user should be in the shared directory. If the file is not in the shared directory, modify the ~/.bashrc file of each node.

  6. Run the following command for the settings to take effect:

    source ~/.bashrc