Boot Latency Optimization

Add the following options to the mpirun command to optimize the boot process:

• --mca osc ^ucx: The UCX is not specified when the unilateral communication module (OSC) is used. Since UCX initialization adds significant latency and most collective communications rely on PML rather than OSC, this option avoids redundant UCX initialization.
• --mca pml ucx: The PML module initializes only UCX to reduce the impact of initialization of other modules on the boot latency.
• --mca oob_tcp_if_include NIC name: To accelerate the osu_init fence process, you are advised to use NICs with high TCP bandwidth and low latency. This is because fence requires inter-node TCP communication, and the out-of-band TCP communication NIC often becomes the performance bottleneck during service initialization. By default, the first NIC obtained in the environment is used. If there are other NICs with better performance, you can set this option to reduce communication latency.
• -x UCX_TLS=sm,rc_v: The combination of sm and rc_v functions as the transport layer.
  • sm indicates that the sm shared memory is used as the transport layer on a node. It accelerates the communication of processes on the node.
  • Between nodes, using rc_v instead of rc speeds up service boot because rc_v only needs rc_v and ud_v initialized, whereas rc also requires initializing rc_mlx and ud_mlx.
• -x UCX_NET_DEVICES=NIC name: The number of selected NICs affects the ud_v initialization latency. Initializing each NIC requires retrieving all its gid_attrs, which is time-consuming. Therefore, you are advised to reduce the number of NICs.

Example command for boot latency optimization:

mpirun --mca osc ^ucx --mca pml ucx -np 32 -x UCX_TLS=sm,rc_v --hostfile file -x UCX_NET_DEVICES=mlx5_0:1 --mca oob_tcp_if_include enp125s0f0 -x PATH -x LD_LIBRARY_PATH osu_init