Documentation
Resource
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Running an MPI Job

This section describes how to run an MPI job.

Background

The collective operations supported by Hyper MPI include Allreduce, Bcast, Barrier, Alltoallv, Allgatherv, Gatherv, and Scatterv. Hyper MPI supports the following algorithm modes.

  • Default algorithm mode: The coll-ucg module is used to run MPI jobs. You do not need to add --mca coll ^ucg when running jobs.
  • Specified algorithm mode: The original collective communication mode of Open MPI is used to run MPI jobs. You need to add --mca coll ^ucg when running jobs.

The following is an example of the specified algorithm mode:

mpirun -np 16 -N 2 --hostfile hf --mca coll ^ucg test_case

  • -np indicates the total number of processes on all nodes. The calculation formula is as follows: -N x Number of nodes.
  • -N indicates the number of processes per node.
  • --hostfile indicates the configuration file of the node where a process is running.
  • test_case indicates the binary file compiled by mpicc.

By default, Hyper MPI binds service processes to cores. If the service process is a multi-threaded program, you need to manually modify the core binding configuration based on the service thread scale to ensure normal thread scheduling. For example:

  • You can specify the rankfile to implement fine-grained core binding. The rules for writing the rankfile are as follows:

    rank 0={ip1} slot=0-7

    rank 1={ip2} slot=0-7

    The preceding rules indicate that MPI task 0 is bound to CPU cores 0 to 7 on node ip1, and MPI task 1 is bound to CPU cores 0 to 7 on node ip2.

    An example command is as follows:

    mpirun --rankfile rf test_case

The --rankfile option cannot be used with --hostfile or -N.

  • You can add --bind-to core, --bind-to numa, or --bind-to socket to implement coarse-grained core binding. An example command is as follows:

    mpirun -np 2 -N 1 --hostfile hf --bind-to numa test_case

Using Default Algorithms

In default algorithm mode, Hyper MPI selects algorithms based on the packet length, PPN, and node quantity. You can directly run the mpirun command without adding extra parameters.

  • The default algorithm for Allreduce, Bcast, and Allgatherv collective operations is selected based on the packet length, PPN, and node quantity.
  • The default algorithm for Barrier, Gatherv, and Scatterv collective operations is selected based on the PPN and node quantity.
  • The default algorithm for Alltoallv collective operations is Ladd.

An example command is as follows:

mpirun -np 16 -N 2 --hostfile hf test_case

Not all algorithms in Hyper MPI support all scenarios. In some scenarios where certain algorithms are not supported, these algorithms are rolled back to the supported ones. For details, see Feature Support and Rollback.

Using Specified Algorithms

The algorithm parameter UCG_PLANC_UCX_ALLREDUCE_ATTR of Allreduce is used as an example. For details about the command parameters of specified algorithms for running an MPI job, see Command Parameters Supported by Specified Algorithms.

  • Method 1: Specify an algorithm using CLI parameters.
    1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
    2. Add the following parameters to the mpirun command:

      -x UCG_PLANC_UCX_ALLREDUCE_ATTR=I:id[S:scoreR:start-end]

  • Method 2: Select an algorithm by setting environment variables.
    1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
    2. Write the following command to the ~/.bashrc file on all nodes:

      export UCG_PLANC_UCX_ALLREDUCE_ATTR=I:id[S:scoreR:start-end]

      • id indicates the sequence number of an algorithm. For details about the sequence number of the corresponding algorithm, see Algorithms. If the selected value is not within the valid range, the default algorithm is executed.
      • score indicates the score of an algorithm. A higher score indicates a higher priority. If this parameter is not specified, the algorithm has the highest priority by default. The value can be any natural number.
      • start-end indicates the range of the packet size that the algorithm applies to. If start-end is not specified, the algorithm applies to all packet sizes by default. The values of start and end can be any natural number.
    3. Make environment variables take effect.

      source ~/.bashrc

    4. If the environment variables are no longer needed, perform the following operations:

      Delete the export UCG_PLANC_UCX_ALLREDUCE_ATTR=I:id[S:scoreR:start-end] command written in ~/.bashrc.

      Run the following command to delete the environment variables that have taken effect:

      unset UCG_PLANC_UCX_ALLREDUCE_ATTR

Command Parameters Supported by Specified Algorithms

  • Open MPI command parameters compatible with Hyper MPI

    For details, see https://www.open-mpi.org/doc/current/man1/mpirun.1.php.

  • New command parameters of Hyper MPI

    Table 1 lists the common command parameters for running MPI jobs. For more command parameters, see Command Parameters.

    Table 1 Common command parameters supported by the specified algorithms

    Hyper MPI Command Parameter

    Value

    Description

    -x UCG_PLANC_UCX_BCAST_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 6 or from 8 to 14.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Bcast algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_BCAST_ATTR=I:4

    -x UCG_PLANC_UCX_ALLREDUCE_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 8 or from 11 to 14.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Allreduce algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_ALLREDUCE_ATTR=I:7

    -x UCG_PLANC_UCX_BARRIER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 7 or can be 10.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Barrier algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_BARRIER_ATTR=I:1

    -x UCG_PLANC_UCX_ALLTOALLV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 5.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Alltoallv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_ALLTOALLV_ATTR=I:1

    -x UCG_PLANC_UCX_ALLGATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 6.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Allgatherv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_ALLGATHERV_ATTR=I:3

    -x UCG_PLANC_UCX_SCATTERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 3.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Scatterv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_SCATTERV_ATTR=I:1

    -x UCG_PLANC_UCX_GATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Gatherv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_GATHERV_ATTR=I:1

    -x UCG_PLANC_UCX_GATHER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Gather algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_GATHER_ATTR=I:1

    -x UCG_PLANC_STARS_BCAST_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Bcast offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_BCAST_ATTR=I:1

    -x UCG_PLANC_STARS_ALLGATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Allgatherv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_ALLGATHERV_ATTR=I:1

    -x UCG_PLANC_STARS_SCATTERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Scatterv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_SCATTERV_ATTR=I:1

    -x UCG_PLANC_STARS_ALLTOALLV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Alltoallv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_ALLTOALLV_ATTR=I:1

    -x UCG_PLANC_STARS_BARRIER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Barrier offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_BARRIER_ATTR=I:1

    -x UCG_PLANC_UCX_IBCAST_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 6 or from 8 to 14.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IBcast algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IBCAST_ATTR=I:4

    -x UCG_PLANC_UCX_IALLREDUCE_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 8 or from 11 to 14.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IAllreduce algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IALLREDUCE_ATTR=I:7

    -x UCG_PLANC_UCX_IBARRIER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 7 or can be 10.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IBarrier algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IBARRIER_ATTR=I:1

    -x UCG_PLANC_UCX_IALLTOALLV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 5.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IAlltoallv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IALLTOALLV_ATTR=I:1

    -x UCG_PLANC_UCX_IALLGATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 6.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IAllgatherv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IALLGATHERV_ATTR=I:3

    -x UCG_PLANC_UCX_ISCATTERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 3.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IScatterv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_ISCATTERV_ATTR=I:1

    -x UCG_PLANC_UCX_IGATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IGatherv algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IGATHERV_ATTR=I:1

    -x UCG_PLANC_UCX_IGATHER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IGather algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IGATHER_ATTR=I:1

    -x UCG_PLANC_STARS_IBCAST_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value ranges from 1 to 4.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IBcast offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_IBCAST_ATTR=I:1

    -x UCG_PLANC_STARS_IALLGATHERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IAllgatherv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_IALLGATHERV_ATTR=I:1

    -x UCG_PLANC_STARS_ISCATTERV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IScatterv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_ISCATTERV_ATTR=I:1

    -x UCG_PLANC_STARS_IALLTOALLV_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IAlltoallv offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_IALLTOALLV_ATTR=I:1

    -x UCG_PLANC_STARS_IBARRIER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value is 1.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IBarrier offloading algorithm parameter.

    Example:

    -x UCG_PLANC_STARS_IBARRIER_ATTR=I:1

    -x UCG_PLANC_UCX_REDUCE_SCATTER_BLOCK_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value can be 1 or 2.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Reduce_scatter_block algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_REDUCE_SCATTER_BLOCK_ATTR=I:1

    -x UCG_PLANC_UCX_IREDUCE_SCATTER_BLOCK_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value can be 1 or 2.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IReduce_scatter_block algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IREDUCE_SCATTER_BLOCK_ATTR=I:1

    -x UCG_PLANC_UCX_REDUCE_SCATTER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value can be 1 or 2.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional Reduce_scatter algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_REDUCE_SCATTER_ATTR=I:1

    -x UCG_PLANC_UCX_IREDUCE_SCATTER_ATTR=
    • Value type: string. The format is I:id[S:scoreR:start-end].
    • Value range:
      • id: mandatory. The value can be 1 or 2.
      • score: optional. It can be any natural number.
      • start: optional. It can be any natural number.
      • end: optional. It can be any natural number.
    • Default value: not fixed. Adjust the value based on the effective rules.

    This is an optional IReduce_scatter algorithm parameter.

    Example:

    -x UCG_PLANC_UCX_IREDUCE_SCATTER_ATTR=I:1
Parent topic: Operation Guide
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details