Node Suspension During Collective Communication (i)osu_alltoall(v)/(i)osu_scatter(v)/(i)osu_gather(v) Running on Full Cores

Symptom

When (i)osu_alltoall(v), (i)osu_scatter(v), or (i)osu_gather(v) is running on full cores and the default max_message_size (1 MB) is used, the node is suspended.

Possible Cause

Running collective communication needs more memory than the node has available. The following table shows the memory usage of different collective communications in different OSU versions.

Figure 1 Memory usage in different OSU versions

• PPN: number of processes per node

• np: total number of processes

• max_msg_size: maximum collective communication packet length

Recovery Procedure

Calculate the value of max_msg_size based on the actual memory of a single node and the memory usage formula for (i)osu_alltoall(v), (i)osu_scatter(v), and (i)osu_gather(v) in different OSU versions. Use the -m option in the mpirun command to specify the maximum packet length.