NIC Driver Error "Transport Retry Counter Exceeded" Reported During MPI Job Running

Symptom

When an MPI job is running, the NIC driver reports the following error:

rc_verbs_iface.c:120 send completion with error: transport retry counter exceeded [qpn 0x272 wrid 0x1 vendor_err 0x15]

Possible Cause

The transport layer RC in HUCX relies on the NIC hardware for retries instead of using software retries. When the transport layer is initialized, HUCX sets the default RC timeout interval and retry count. If the data transmission in the cluster is slow and exceeds the total timeout interval, the NIC returns a timeout event to HUCX. Upon receiving this event, HUCX exits the application.

Recovery Procedure

• Increase the timeout interval for a single RC request by setting the UCX_RC_TIMEOUT parameter. The default value is 1s.
• Increase the number of retransmission times upon RC timeout by setting the UCX_RC_RETRY_COUNT parameter. The default value is 7.

Set the preceding two parameters based on the actual network conditions.

• A high value may delay NIC fault detection and consume system resources.

• A low value may cause NIC timeout, application instability, and performance deterioration in some scenarios (for example, network congestion).

• Increase the value slowly if a timeout happens and check the results. The right value should be slightly above the maximum network latency.