Feature Support and Rollback
Not all algorithms in Hyper MPI support all scenarios. The following scenarios are supported only by part of algorithms:
Scenario 1: Unbalanced Processes Per Node (PPN)
If PPN is not specified or the rankfile is used when the mpirun command is used to submit tasks, node processes will be unbalanced. Allreduce algorithm 11, 13, or 14, Allgatherv algorithm 6, Scatterv algorithm 3, Barrier algorithm 10, or Alltoallv algorithm 2 will be automatically rolled back to a supported algorithm.
Scenario 2: Unbalanced Processes Per Socket (PPS)
If PPS is not specified or rankfile is used when the mpirun command is used to submit tasks, socket processes will be unbalanced. Allreduce algorithm 13 or 14 will be automatically rolled back to a supported algorithm.
Scenario 3: Non-Commutative Operations
The MPI standard allows users to define mathematical operations in Allreduce in addition to built-in mathematical operations such as summation (MPI_SUM) and product (MPI_PROD). User-defined operations such as matrix multiplication may be non-commutative. Allreduce algorithm 1 supports these operations.
If the specified algorithm is not algorithm 1, Hyper MPI rolls back the algorithm to algorithm 1 (for small- and medium-sized packets) based on the data packet size.
Scenario 4: --bind-to none or --bind-to board
If --bind-to is set to none or board, Allreduce algorithm 3, 6, 8, 13, or 14, or Barrier algorithm 3, 5, or 7 will be automatically rolled back to a supported algorithm.
Scenario 5: Number of Sent Messages (count) Less Than Number of Processes on All Nodes (np) or Number of Processes on All Nodes (np) Minus 1
np indicates the total number of MPI processes specified by the mpirun -np parameter. It is calculated as follows: np = Number of nodes x PPN.
If the number of sent messages (count) is less than the number of processes on all nodes (np), Allreduce algorithm 4, 12, 13, or 14, or Bcast algorithm 8 or 11 is not supported and will be automatically rolled back to a supported algorithm.
If the number of sent messages (count) is less than the number of processes on all nodes (np) minus 1, Bcast algorithm 14, or Bcast offloading algorithm 3 or 4 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 6: Remainder of count Divided by ppn Is Not 0
If the remainder of count divided by ppn is not 0, Allreduce algorithm 13 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 7: PPS Is 1
If PPS is 1, Allreduce algorithm 14 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 8: PPN Is 1
If PPN is 1, Allreduce algorithm 11, 13, or 14, Allgatherv algorithm 6, Scatterv algorithm 3, Barrier algorithm 10, or Alltoallv algorithm 2 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 9: np Is Not 2 to the Power of N
If np is not 2 to the power of N, Allreduce algorithm 11 or Barrier algorithm 10 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 10: np Is an Odd Number
If np is an odd number, Allgatherv algorithm 1 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 11: --rank-by node or --rank-by board
If --rank-by is set to node or board, Alltoallv algorithm 2 or Scatterv algorithm 3 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 12: Number of Nodes Is 1
If the number of nodes is 1, Alltoallv algorithm 2, Allgatherv algorithm 6, or Scatterv algorithm 3 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 13: sendbuf Is MPI_IN_PLACE
If sendbuf is MPI_IN_PLACE, all Alltoallv algorithms and Scatterv algorithm 3 are not supported and will be rolled back to Open MPI.
Scenario 14: send or recv total size Exceeds 2^32 - 1 in Alltoallv Collective Operations
In this scenario, Alltoallv algorithm 2 (node-aware plummer) is runtime data and cannot be judged during the rollback. As a result, an error message "Alltoallv node-aware plummer don't support datasize >= 2^32 bytes" is displayed.
Scenario 15: Sum of sendcounts of All Processes on a Node Exceeds 2^31 in Alltoallv Collective Operations
In this scenario, Alltoallv algorithm 2 (node-aware plummer) is runtime data and cannot be judged during the rollback. As a result, an error message "Alltoallv node-aware plummer gatherv phase don't support displs >= 2^31 bytes" is displayed. The sum of sendcounts is calculated as follows:
p indicates the total number of processes, ppn indicates the number of processes per node, and sendcounts[i] indicates the sendcounts array of the ith process on the node.
Scenario 16: Inconsistent sendcounts and recvcounts for Each Process in Alltoallv Collective Operations
In this scenario, if no Alltoallv algorithm is specified in the command parameters, processes may roll back to different algorithms. As a result, the program cannot run properly. If the sendcounts and recvcounts values are the same for each process, you do not need to specify the Alltoallv algorithm. The algorithm with the optimal performance is selected by default.
Scenario 17: Number of Processes Less Than 3
If the number of processes is less than 3, Bcast offloading algorithm 1 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 18: Number of Processes Less Than 4
If the number of processes is less than 4, Bcast offloading algorithm 2 is not supported and will be automatically rolled back to a supported algorithm.
Scenario 19: USE_SHM_POOL Disabled for the Alltoallv plummer-opt Algorithm
If the plummer-opt algorithm is specified but UCG_USE_SHM_POOL is not set to y, the system automatically rolls back to a supported algorithm.