HUCX RC Multi-Port Failover

Hyper MPI HUCX supports reliably-connected (RC) multi-port failover (rc_failover). This feature enables multi-port failover by automatically detecting port failures and rerouting traffic to available ports, minimizing service interruptions.

Currently, this feature supports only the RC connection mode. You can configure this feature by adding the options listed in Table 1 to the mpirun command.

**Table 1** Option configuration
Option	Required (Yes/No)	Description
-x UCX_TLS=rc_f	Yes	Sets the transport mode to rc_f.
-x UCX_MAX_EAGER_RAILS=2	Yes	Uses the EAGER multi-rail mode. The EAGER channel is usually used for small packets and bilateral communication. The value is an integer ranging from 1 to 4. The default value is 1. To enable this function, set the value to an integer greater than 1. The value can be determined based on the NIC fault rate of the node. Higher values mean higher reliability. The recommended value is 2.
-x UCX_LOCAL_CONN_SAME_DEV=y	Yes	Determines whether to use NICs with the same name to establish links on the local node. The value can be y or n. The default value is n. y: NICs with the same name are used to establish links on the local node, which can narrow down the fault domain. n: Different processes on the local node use NUMA-affinity NICs to establish links.
-x UCX_MAX_RNDV_RAILS=2	No	Uses the RNDV multi-rail mode. The RNDV channel is usually used for large packets and unilateral communication. The value is an integer ranging from 2 to 4. The default value is 2. The value can be determined based on the NIC fault rate of the node. Higher values mean higher reliability. The recommended value is 2.
-x UCX_MAX_AUX_RAILS=2	No	Uses the AUX multi-rail mode. During initialization in RC mode, HUCX employs AUX links to exchange addresses and establish RC links. These AUX links typically utilize Unreliable Datagram (UD) connections. For large-scale service where initialization link establishment time is prolonged, it is advised to use multi-rail AUX links and increase the number of AUX rails. This mitigates the impact of network port failures on the services during the initialization link establishment process. The value is an integer ranging from 1 to 4. The default value is 1. The value can be determined based on the NIC fault rate of the node. Higher values mean higher reliability. The recommended value is 2.
-x UCX_AUX_TIMEOUT=3s	No	Sets the AUX multi-rail link setup timeout period. The default value is 20s. Format: -x UCX_AUX_TIMEOUT=<number>[s\|us\|ms\|ns] The value of <number> is an integer greater than 0. For large-scale clusters and workloads, extended connection times are expected. Adjust this value based on your cluster scale and workloads to ensure timely failover triggering.
-x UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1	No	Sets the range of network ports to be selected. By default, all network ports are allowed.

HUCX multi-port failover is limited to the selected ports within a single service process. If the number of selected ports exceeds the configured rail count, the ports are selected according to the rail limit, leaving some ports unselected.

Automatic failover redirects traffic to available ports on port failures. The failback mechanism is non-revertive for active processes, meaning restored ports will only be utilized by newly initiated processes.

If a port fails and no other port is available, the service will exit with an error.

Example command:

mpirun -np 64 -N 32 --hostfile hf -x UCX_TLS=rc_f -x UCX_MAX_EAGER_RAILS=2 -x UCX_LOCAL_CONN_SAME_DEV=y -x UCX_MAX_AUX_RAILS=2 -x UCX_AUX_TIMEOUT=3s osu_bcast

Parent topic: Command Description and Example