Configuring Huge Pages for the Data Segment

Configuring huge pages for the data segment can reduce the probability of translation lookaside buffer (TLB) miss and CPU scheduling caused by insufficient pages, thereby improving write performance. Perform the following steps on all server nodes.

1. Configure system huge pages.

   1	echo 163840 > /proc/sys/vm/nr_hugepages

   

   • On openEuler 20.03, the 64 KB kernel page has a bug. You need to recompile the kernel to change the memory page size to 4 KB. Run the following command to check the kernel page size:

     1	getconf PAGESIZE

     

     If the command output is 4096, the kernel page size is 4 KB. If not, perform compilation and installation by following instructions in "Compiling the Kernel" in the Bcache User Guide.

   • The memory capacity occupied by system huge pages must be greater than or equal to the memory required by OSDs on the current node (osd_memory_target x Number of OSDs). For example, if each node is deployed with 24 OSDs and osd_memory_target is set to 10 GB, the hugepage memory must be at least 240 GB. In the 4 KB kernel page environment, the size of a huge page is 2 MB. Therefore, the number of huge pages must be at least 120,000.
   • If the system has been running for a long time, the memory may be too fragmented to allocate a certain number of huge pages. In this case, you can add the hugepage allocation options to the configurations of the corresponding kernel (run the uname -r command to query) and reboot the server to validate the modification.
     1. Open the grub.cfg file and add hugepagesz=2M and hugepages=163840 to the file.

        1	vim /boot/efi/EFI/openEuler/grub.cfg

        

     2. Reboot the node and query hugepage usage to check whether the hugepage configuration has taken effect.

        1 2	reboot cat /proc/meminfo | grep Huge

        

2. Enable automatic mounting of the hugepage memory.
   1. Create a directory for mounting huge pages.

      1	mkdir /mnt/huge

   2. Add the hugepage configuration to the fstab file.

      1	vim /etc/fstab

      Add the following configuration:

      1	nodev /mnt/huge hugetlbfs defaults,pagesize=2M 0 0

      

3. Modify the Ceph service and enable the TCMalloc hugepage configuration.

   1	vim /usr/lib/systemd/system/ceph-osd@.service

   Add the following environment variable.

   1	TCMALLOC_MEMFS_MALLOC_PATH=/mnt/huge/osd%i

   

4. Reload configurations.

   1	systemctl daemon-reload

5. Restart the OSD process.

   1	systemctl restart ceph-osd.target