Transport Mode Selection

Run the following command to view the network devices and supported transport modes:

ucx_info -d

Command output example:

#   Transport: ud_verbs 
#      Device: mlx5_0:1

Device indicates the current network device, and Transport indicates the transport mode supported by the current network device. Multiple transport modes are allowed.

When starting an MPI job, you can set the transport mode used by MPI by using the -x UCX_TLS option. Currently, common parameters include ud, rc, shm, mm, rc_x, ud_x, and dc_x. You can select either shm or mm, and one or more of ud, rc, rc_x, ud_x, and dc_x. If the option is not set, the default setting -x UCX_TLS=all is used, indicating that all transport modes supported by the current network device are used.

For more transport modes and their meanings, see https://github.com/openucx/ucx/wiki/UCX-environment-parameters.

The default settings may not be optimal if no scenario is specified. To achieve optimal performance, you can try the following combinations:

-x UCX_TLS=shm,rc_x,ud_x

-x UCX_TLS=shm,ud_x

-x UCX_TLS=shm,dc_x

It is recommended that the ud_x and shm transport modes be used in the Mellanox NIC environment, the ud and shm transport modes be used in the Kunpeng RoCE NIC environment. The reason is that the communication is faster in ud mode and the number of QPs created in ud mode is fewer than that in rc mode, and the ud_x mode is not supported in the Kunpeng RoCE NIC environment.