UD Timeout Error Caused by Network Issues Including Packet Loss: UD Endpoint... unhandled timeout error

Symptom

The following error is reported when an MPI job runs:

[arm-129:435170:0:435170]       ud_ep.c:262  Fatal: UD endpoint 0x6c29690 to <no debug data>: unhandled timeout error 
==== backtrace (tid: 435170) ==== 
 0  /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(ucs_handle_error+0x250) [0x4000237b3630] 
 1  /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(ucs_fatal_error_message+0xd0) [0x4000237b0940] 
 2  /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(+0x1fa08) [0x4000237b0a08] 
 3  /workspace/cw/ccsuite/hmpi/install/hucx/lib/ucx/libuct_ib.so.0(+0x475a8) [0x4000238795a8] 
 4  /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(+0x18d10) [0x4000237a9d10] 
 5  /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucp.so.0(ucp_worker_progress+0x60) [0x4000236f47b0] 
 6  /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libopen-pal.so.40(opal_progress+0x38) [0x4000223cbef8] 
 7  /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libmpi.so.40(ompi_mpi_init+0xc78) [0x4000220be608] 
 8  /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libmpi.so.40(MPI_Init+0x64) [0x400022066404] 
 9  /workspace/cw/cwScript/mpijob/bcast_sleep_accurate() [0x400a6c] 
10  /usr/lib64/libc.so.6(+0x2afbc) [0x400022146fbc] 
11  /usr/lib64/libc.so.6(__libc_start_main+0x94) [0x400022147094] 
12  /workspace/cw/cwScript/mpijob/bcast_sleep_accurate() [0x400930] 
================================= 
[arm-129:435170] *** Process received signal *** 
[arm-129:435170] Signal: Aborted (6) 
[arm-129:435170] Signal code:  (-6) 
[arm-129:435170] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x400021fd393c] 
[arm-129:435170] [ 1] /usr/lib64/libc.so.6(+0x80e78)[0x40002219ce78] 
[arm-129:435170] [ 2] /usr/lib64/libc.so.6(raise+0x1c)[0x400022158cfc] 
[arm-129:435170] [ 3] /usr/lib64/libc.so.6(abort+0xe0)[0x400022146d2c] 
[arm-129:435170] [ 4] /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(+0x1f944)[0x4000237b0944] 
[arm-129:435170] [ 5] /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(+0x1fa08)[0x4000237b0a08] 
[arm-129:435170] [ 6] /workspace/cw/ccsuite/hmpi/install/hucx/lib/ucx/libuct_ib.so.0(+0x475a8)[0x4000238795a8] 
[arm-129:435170] [ 7] /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucs.so.0(+0x18d10)[0x4000237a9d10] 
[arm-129:435170] [ 8] /workspace/cw/ccsuite/hmpi/install/hucx/lib/libucp.so.0(ucp_worker_progress+0x60)[0x4000236f47b0] 
[arm-129:435170] [ 9] /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libopen-pal.so.40(opal_progress+0x38)[0x4000223cbef8] 
[arm-129:435170] [10] /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libmpi.so.40(ompi_mpi_init+0xc78)[0x4000220be608] 
[arm-129:435170] [11] /workspace/cw/ccsuite/hmpi/install/hmpi/lib/libmpi.so.40(MPI_Init+0x64)[0x400022066404] 
[arm-129:435170] [12] /workspace/cw/cwScript/mpijob/bcast_sleep_accurate[0x400a6c] 
[arm-129:435170] [13] /usr/lib64/libc.so.6(+0x2afbc)[0x400022146fbc] 
[arm-129:435170] [14] /usr/lib64/libc.so.6(__libc_start_main+0x94)[0x400022147094] 
[arm-129:435170] [15] /workspace/cw/cwScript/mpijob/bcast_sleep_accurate[0x400930] 
[arm-129:435170] *** End of error message *** 
-------------------------------------------------------------------------- 
mpirun noticed that process rank 5 with PID 435171 on node arm-129 exited on signal 6 (Aborted). 
--------------------------------------------------------------------------

Possible Cause

• Using unpatched HUCX source code with UCX_TLS=ud set through the scheduler causes timeout errors when an MPI job is suspended and subsequently resumed.
• The RoCE network is used, but the lossless network is not configured on the NIC and switch.
• The network between compute nodes is faulty.

Recovery Procedure

1. This issue is resolved in the latest HUCX patch. Download the patch for your version and rebuild to install.
2. If the lossless network is not configured on the NIC and switch, configure the lossless network and then run the job.
3. If the lossless network is normal, check whether the network between the nodes where the error occurs is normal.
4. If no problem is found on the physical link and hardware configuration, you can set -x UCX_UD_TIMER_BACKOFF=1 -x UCX_UD_TIMER_TICK=100ms -x UCX_UD_TIMEOUT=600s to increase the timeout interval.