Too Many Job Processes: There are not enough slots available...

Symptom

The mpirun command fails to be executed because the number of processes that submit MPI jobs is greater than the total number of CPU cores on job execution nodes in the cluster.

The following is an example of the execution failure:

mpirun -np 1025 --hostfile hf8 hmpifile_2021/allreduce/AllReduce

-------------------------------------------------------------------------- 
There are not enough slots available in the system to satisfy the 1025 
slots that were requested by the application: 
  
  hmpifile_2021/allreduce/AllReduce 
  
Either request fewer slots for your application, or make more slots 
available for use. 
  
A "slot" is the Open MPI term for an allocatable unit where we can 
launch a process.  The number of slots available are defined by the 
environment in which Open MPI processes are run: 
  
  1. Hostfile, via "slots=N" clauses (N defaults to number of 
     processor cores if not provided) 
  2. The --host command line parameter, via a ":N" suffix on the 
     hostname (N defaults to 1 if not provided) 
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.) 
  4. If none of a hostfile, the --host command line parameter, or an 
     RM is present, Open MPI defaults to the number of processor cores 
  
In all the above cases, if you want Open MPI to default to the number 
of hardware threads instead of the number of processor cores, use the 
--use-hwthread-cpus option. 
  
Alternatively, you can use the --oversubscribe option to ignore the 
number of available slots when deciding the number of processes to 
launch. 
--------------------------------------------------------------------------

1025 indicates the number of processes that submit MPI jobs.

Possible Cause

During mpirun execution, the number of processes is greater than the total number of CPU cores on the job execution nodes in the cluster.

Recovery Procedure

1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
2. Run the following command to query the number of CPU cores of each job execution node:

   lscpu

   Architecture:          aarch64 
   Byte Order:            Little Endian 
   CPU(s):                128 
   On-line CPU(s) list:   0-127 
   Thread(s) per core:    1 
   Core(s) per socket:    64 
   Socket(s):             2 
   NUMA node(s):          4 
   Model:                 0 
   CPU max MHz:           2600.0000 
   CPU min MHz:           200.0000 
   BogoMIPS:              200.00 
   L1d cache:             64K 
   L1i cache:             64K 
   L2 cache:              512K 
   L3 cache:              65536K 
   NUMA node0 CPU(s):     0-31 
   NUMA node1 CPU(s):     32-63 
   NUMA node2 CPU(s):     64-95 
   NUMA node3 CPU(s):     96-127 
   Flags:                 fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop

3. Calculate the total number of CPU cores on the job execution nodes in the cluster and ensure that the number of processes is fewer than or equal to the total number of CPU cores during mpirun execution. Assume that there are eight nodes with 1,024 CPU cores. Run the following command to submit an MPI job:

   mpirun -np 1024 --hostfile hf8 hmpifile_2021/allreduce/AllReduce

   All tests are success