Slow Hyper MPI Boot Due to Large Open Files Value

Symptom

The value of open files in ulimit is too large (for example, 1073741816). This causes the mpirun command to hang for three minutes when Hyper MPI runs tasks across multiple nodes.

Possible Cause

Before running SSH, you need to traverse and close all file descriptors. If the value of open files set by running the ulimit -n command is too large, the Hyper MPI boot latency increases.

Recovery Procedure

1. Check whether the Hyper MPI boot latency is long due to a large value of fdmax.

   Add --mca plm_base_verbose 11 to the command to enable the log function. Search for the plm:rsh: close max fd field in the printed content to check the current value of fdmax. Generally, when the value of fdmax is greater than 1073741816, the boot latency is longer than 3 minutes. A larger value indicates a longer latency.

2. If the Hyper MPI boot latency is long due to a large value of fdmax, run the ulimit -n open files value command to change the value of open files as required. The value of fdmax is the smaller one between the maximum value of the file descriptor and the value of open files. Then, check whether the new value of fdmax takes effect by referring to Step 1.