UCP Request Timed Out
Symptom
During the test of OSU collective communication operations osu_iallgather and osu_iallgatherv, when the size is greater than 32768, the UCP request times out.
Possible Cause
Running the command with Open MPI 4.1.1 and Open UCX 1.15.0 causes the same issue, confirming the issue lies within these open-source components. This may be due to packet fragmentation caused by bcopy, combined with an insufficient default receive queue length, which leads to packet drops.
Recovery Procedure
Add -x UCX_RNDV_THRESH=8192 to the running command.
Parent topic: FAQ