System Tuning

Optimizing the OS Configuration

Purpose
Adjust the system configuration to maximize the hardware performance.

Procedure

Table 1 lists the optimization items.

Table 1 OS configuration parameters

Parameter

Description

Suggestion

Configuration Method

vm.swappiness

The swap partition is the virtual memory of the system. It is not recommended because it will degrade system performance.

Default value: 60

Symptom: The performance deteriorates significantly when the swap partition is used.

Suggestion: Disable the swap partition and set this parameter to 0.

Run the following command:

sudo sysctl vm.swappiness=0

MTU

Maximum size of a data packet allowed by a NIC. After the value is increased, the number of network packets can be reduced and the efficiency can be improved.

Default value: 1500 bytes

Symptom: Run the ip addr command to view the value.

Suggestion: Set the MTU size to 9000 bytes.

Run the following command:
1
vim /etc/sysconfig/network-scripts/ifcfg-$(Interface)
Add MTU="9000".
NOTE:
${Interface} indicates the interface name.
After the configuration is complete, restart the network service.
1
service NetworkManager restart

pid_max

A tunable parameter provided by the kernel, which is used to limit the maximum number of processes that can be allocated by the system. The default value of pid_max is 32768, which is sufficient in normal cases. However, when heavy workloads are being processed, this value may be insufficient and cause memory allocation failure.

Default value: 32768

Symptom: Run the cat /proc/sys/kernel/pid_max command to view the value.

Suggestion: Set the maximum number of threads that can be generated in the system to 4194303.

Run the following command:

echo 4194303 > /proc/sys/kernel/pid_max

file_max

Maximum number of files that can be opened by all processes in the system. In addition, some programs can call setrlimit to set the limit on each process. If the system generates a large number of errors indicating that file handles are used up, increase the value of this parameter.

Default value: 13291808

Symptom: Run the cat /proc/sys/fs/file-max command to view the value.

Suggestion: Set the value of this parameter to the one queried by running the cat /proc/meminfo | grep MemTotal | awk '{print $2}' command.

Run the following command:

echo ${file-max} > /proc/sys/fs/file-max

NOTE:

${file-max} indicates the value displayed after cat /proc/meminfo | grep MemTotal | awk '{print $2}' is run.

read_ahead

Linux readahead means that the Linux kernel prefetches a certain area of the specified file and loads it into the page cache. As a result, when the area is accessed subsequently, blocks caused by page faults will not occur.

Reading data from memory is much faster than from drives. Therefore, the readahead feature can effectively reduce the number of drive seeks and the I/O waiting time of applications. It is one of the important methods for optimizing the drive read I/O performance.

Default value: 128 KB

Symptom: Run the /sbin/blockdev --getra /dev/sdb to view the value.

Suggestion: Change the value to 8192 KB. This improves the drive read efficiency by pre-reading and recording data to random access memory (RAM).

Run the following command:

/sbin/blockdev --setra <sectors> /dev/sdb  

NOTE:

/dev/sdb is used as an example. <sectors> indicates the readahead size, and you are advised to set it to 8192. You need to modify this parameter for all data drives.

I/O_Scheduler

The Linux I/O scheduler is a component of the Linux kernel. You can adjust the scheduler to optimize system performance.

Default value: CFQ

Symptom: The Linux I/O scheduler needs to be configured based on storage devices to achieve the optimal system performance.

Suggestion: Set the I/O scheduling policy to deadline for HDDs and none for SSDs.

Run the following command:

echo deadline > /sys/block/sdb/queue/scheduler
echo none > /sys/class/block/nvme0n1/queue/scheduler

NOTE:

/dev/sdb and /dev/nvme0n1 are used as an example. You need to modify this parameter for all data drives.

nr_requests

A block device I/O scheduling parameter provided by the kernel. It controls the I/O request queue depth of each block device (such as an HDD or SSD). If the Linux system receives a large number of read requests, the default number of request queues may be insufficient. To deal with this problem, you can dynamically adjust the default number of request queues in the /sys/block/hda/queue/nr_requests file.

Default value: 128 (1023 for SSDs)

Symptom: Adjusting the nr_requests parameter can increase the drive throughput.

Suggestion: Set the number of drive request queues to 512 (1023 for SSDs).

Run the following command:

echo 512 > /sys/block/sdb/queue/nr_requests
echo 1023 > /sys/block/nvme0n1/queue/nr_requests

NOTE:

/dev/sdb and /dev/nvme0n1 are used as an example. You need to modify this parameter for all data drives.

NUMA Affinity Tuning

Procedure
Evenly allocate network and storage resources to NUMA nodes.

Purpose

In this example, 12 NVMe SSDs and four network ports are evenly allocated to four NUMA nodes.

The NVMe SSD numbers range from 0 to 11, and the network port names are enps0f0, enps0f1, enps0f2, and enps0f3.

for i in {0..11}; do echo `expr ${i} / 3` > /sys/class/block/nvme${i}n1/device/device/numa_node; done
for j in {0..3}; do echo ${j} > /sys/class/net/enps0f${j}/device/numa_node; done

Parent topic: High-Performance Storage