Error "IBV_WR_Complete Fail, Ret 12" Reported When an MPI Job Runs with RC_ACC Stars Transmission Protocol Enabled

Symptom

If an MPI job runs with Stars collective communication offload enabled and the Stars transmission protocol set to rc_acc, the following error message is displayed:

ibv_wr_complete fail, ret 12

Possible Cause

Certain Stars algorithms (like Bcast algorithms 3 and 4, and Allgatherv algorithm 1) send many tasks to the IBV QP. If there are too many job processes, the QP queue may overflow, causing this error.

Recovery Procedure

• Increase the depth of the Stars RC QP queue by setting the UCG_PLANC_STARS_RC_TX_QUEUE_LEN parameter. The default value is 1024. You are advised to set this parameter to 8192.
• It is recommended that the number of processes in a single communication domain be fewer than or equal to 1024 when Stars is used for offload.