Transmission Mode Selection

Run the following command to view the network devices and supported transmission modes:

ucx_info -d

Information similar to the following is displayed:

Transport: ud_verbs
#      Device: mlx5_0:1
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1030 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 7 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 880
#           connection: to ep, to iface
#      device priority: 38
#     device num paths: 1
#              max eps: inf
#       device address: 17 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure

ud_verbs indicates the transmission mode, and mlx5_0:1 indicates the network device.

The -x UCX_TLS command can be used to adjust the transmission mode for MPI. Currently, common parameters in the command include ud, rc, shm, rc_x and ud_x.

For more transmission modes and their meanings, see https://github.com/openucx/ucx/wiki/UCX-environment-parameters.

The default parameters may not be optimal in some scenarios. You can try the following combinations:

-x UCX_TLS=shm,rc_x,ud_x

-x UCX_TLS=shm,ud_x

-x UCX_TLS=shm,dc_x

-x UCX_TLS=mm,rc_x,ud_x

For shm and mm, they cannot be used together. For rc, ud, rc_x, ud_x, and dc_x, you can use one or more of them in the same command.

Not setting this parameter is equivalent to -x UCX_TLS=all by default.

It is recommended that the ud_x and shm transmission modes be used in the Mellanox NIC environment, the ud and shm transmission modes be used in the Kunpeng RoCE NIC environment. The reason is that the communication is faster in ud mode and the number of QPs created in ud mode is fewer than that in rc mode, and the ud_x mode is not supported in the Kunpeng RoCE NIC environment.