Memory Used Up During osu_alltoall, osu_alltoallv, osu_alltoallw, osu_ialltoall, osu_ialltoallv, or osu_ialltoallw Runtime

Symptom

When osu_alltoall, osu_alltoallv, osu_alltoallw, osu_ialltoall, osu_ialltoallv, or osu_ialltoallw is run on 576 cores with about 500 GB of memory and the default max_message_size (1 MB) is used, the memory is used up.

Possible Cause

Tests and analysis show that running osu_alltoall, osu_alltoallv, osu_alltoallw, osu_ialltoall, osu_ialltoallv, or osu_ialltoallw requires a large amount of memory. The memory usage is as follows:

1. Each process requires two memory blocks: sendbuf and recvbuf. The size of each memory block is as follows:

   bufsize = options.max_message_size × numprocs (aligned by page size)

   When using the -c option, you need two extra memory blocks: expected_buffer and temp_buffer. Both must match the size of sendbuf and recvbuf.

2. size_um: memory required for UM communication.

Memory size required by each node:

• If the -c option is not specified:
• If the -c option is specified:

The default value of max_message_size for osu_alltoall is 1 MB. If there are two processors and 575 processes on a single node, at least 650 GB of memory is required, which is far greater than 512 GB (16 × 32 GB DIMMs) on a single node.

Recovery Procedure

Set max_message_size based on the preceding formula. For example, if there are 16 nodes with 576 processes and 500 GB of memory each, you are advised to set -m 16384 to set max_message_size to 16 KB when specifying the -c option.