System Exceptions Caused by Small SWAP Partition Size

Symptom

During a database test, the message "-bash: fork: Cannot allocate memory" is displayed after a basic Linux command is executed.

After the database is restarted, the problem does not occur after the Linux command is executed again. However, the number of database processes decreases in subsequent tests.

Key Process and Cause Analysis

1. In this case, the maximum number of processes in the system is set to 32768, which is very large. Therefore, the probability that the number of database processes reaches the specified limit is low.
2. After the problem occurs, the command for checking the system status cannot be executed. You need to view message logs to locate the fault.
   1. Log in to the BMC, collect and view message logs. It is found that crashes are reported. According to the logs, the system memory is insufficient, and thus some database processes are killed. As a result, the number of database processes decreases.
   2. There is another key message in the logs: The SWAP space has been used up, triggering an out of memory (OOM) error. As a result, system exceptions occur. Compared with previous tests, the memory size used by the database in this test remains the same, but the SWAP space allocated this time (4 GB) is smaller that that allocated in previous tests (150 GB). As a result, OOM is triggered when the memory is insufficient.

Conclusion and Solution

The customer considers that current database memory configuration is optimal and does not want to adjust the database memory. In this regard, dynamically adjust the SWAP partition size to solve the problem.

1. Create an empty file.

   dd if=/dev/zero of=/home/swap bs=1G count=200

2. Mount it to the SWAP space.

   mkswap /home/swap

3. Load the SWAP space.

   /sbin/swapon /home/swap

4. The configurations in 1 to 3 become invalid after the server is restarted. To make the configurations take effect permanently, modify the /etc/fstab configuration file to enable automatic mounting upon system startup. Add the following content to the /etc/fstab file or modify the file as follows:

   /home/swap                       swap                    swap    defaults        0 0