Error "Gatherv Phase Don't Support Displs >= 2^31 Bytes" Reported When Alltoallv Algorithm 2 Is Run

Symptom

The following error is reported when the Alltoallv algorithm 2 (node-aware plummer) is run:

Alltoallv node-aware plummer gatherv phase don't support displs >= 2^31 bytes

Possible Cause

The algorithm is not supported when the sum of sendcounts of all processes on a node exceeds 2^31. For details, see Scenario 15: Sum of sendcounts of All Processes on a Node Exceeds 2^31 in Alltoallv Collective Operations.

For example, if the OSU application runs with 128 processes across 2 nodes, and each node handles 64 processes, the sum of sendcounts of all processes is 128 x 64 x count (count is the value of sendcounts, that is, the number of sent data records. sendcounts in the OSU is the same). When count reaches 256K, the sum of sendcounts is 2G, which exceeds 2^31. In this case, an error log is reported and the job exits.

Recovery Procedure

Send counts must stay within the limits of Alltoallv algorithm 2. If needed, use Alltoallv algorithm 1 (Ladd) instead.