Error Message "Invalid Argument" Reported During QP Creation When an MPI Job Uses the RC or UD Transmission Protocol

Symptom

If an MPI job uses the RC or UD transmission protocol, an error appears when creating the QP.

UCX ERROR Failed to create RC QP TX wr:256 sge:5 in1:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument 
UCX ERROR Failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0: Invalid argument failed: Invalid argument

Possible Cause

BIOS constraints prevent high SGE counts. UCX defaults to 5 SGEs for RC and 6 for UD. Run the ibv_devinfo -v command to view the maximum value of max_sge supported by the current network port.

max_sge:    2

Recovery Procedure

Add -x UCX_RC_VERBS_TX_MIN_SGE=2 -x UCX_UD_VERBS_TX_MIN_SGE=1 to the command.

Set UCX_RC_VERBS_TX_MIN_SGE to the value of max_sge queried by running the ibv_devinfo -v command, and set UCX_UD_VERBS_TX_MIN_SGE to max_sge - 1.