Error Message "No Space Left on Device" Reported During QP Creation When an MPI Job Uses the RC or UD Transmission Protocol

Symptom

If an MPI job uses the RC or UD transmission protocol, an error appears when creating the QP:

UCX ERROR Failed to create RC QP TX wr:256 sge:5 in1:64 resp:64 RX wr:0 sge:1 resp:64: No space left on device failed: No space left on device 
UCX ERROR Failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0: No space left on device failed: No space left on device

Possible Cause

This error occurs when the number of MPI connections exceeds the hardware's maximum QP limit, resulting in resource exhaustion. You can run the ibv_devinfo -v command to view the maximum number of QPs max_qp that can be created on a network port.

max_qp:    1048576

Recovery Procedure

To ensure that MPI jobs can run properly, you can make improvements in the following two aspects:

• Use more network ports for MPI jobs to lower the quantity of QPs needed per port. For example, use two or more network ports for communication.
• Reduce the number of QPs required by MPI jobs. For example, optimize the process layout to reduce connection counts.