UCP Request Timeout Error Due to Network Problems Such As Packet Loss

Symptom

MPI is suspended during running. After the suspension lasts for more than 5 minutes, a timeout warning is displayed, as shown in the following:

UCP request timeout! request tag *** local proc: *** peer proc: *** peer hostname: ***

Possible Cause

• The RoCE network is used, but the lossless network is not configured on the NIC and switch. As a result, packet loss occurs during communication.
• The network between compute nodes is faulty.

Recovery Procedure

1. Find the compute node where the target process handling the timeout request is located from the logs.
2. If the lossless network is not configured on the NIC and switch, configure the lossless network and then run the job.
3. If the lossless network is normal, check whether the network between the nodes where the error occurs is normal.
4. If the network is normal, check the packet loss records on the NIC or switch to determine whether packet loss occurs on the hardware.
5. If no problem is found in the physical links and hardware configuration and the RC protocol is used, use the UD protocol as a temporary solution. Unlike RC, which relies on hardware for retransmission, UD handles retransmissions through software.