FUSE Tuning

Purpose
Optimize the libfuse module to reduce performance loss caused by thread switching between CPU cores and metadata copy.

Preparing the Environment

The libfuse user-mode optimization is developed based on openEuler 22.03 (LTS-SP4). To use this feature, upgrade the Ceph client OS to openEuler 22.03 (LTS-SP4).
openEuler 22.03 provides a built-in libfuse version (3.10.5). libfuse 3.x is supported by the Filesystem in USErspace (FUSE) module of Ceph 16 and later. Therefore, you need to install Ceph 16 or later. Ceph 16.2.7 provided in openEuler 22.03 (LTS-SP4) is recommended. For details, see Ceph 16.2.7 Installation Guide (openEuler 22.03).
The optimized libfuse does not support splice read and write. You need to disable them in the ceph.conf configuration file by modifying related parameters in the global section as follows. For details, see "Configuring MON Nodes" in Ceph File Storage Deployment Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).
fuse_splice_read = false

fuse_splice_write = false

fuse_splice_move = false

ceph-fuse provided in openEuler 22.03 (LTS-SP4) relies on libfuse 2.x. You need to install libfuse3 on openEuler 22.03 (LTS-SP4) and recompile ceph-fuse. The procedure is as follows:

Perform the following steps on either a server or client node.

Install the libfuse3 module.
1
yum install fuse3 fuse3-devel

Go to the /home directory and download Ceph 16.2.7 source code.

cd /home && wget https://download.ceph.com/tarballs/ceph-16.2.7.tar.gz --no-check-certificate

Decompress the source package.
1
tar -zxvf ceph-16.2.7.tar.gz

Install compilation dependencies.

yum install java-devel sharutils checkpolicy selinux-policy-devel gperf cryptsetup fuse-devel gperftools-devel libaio-devel libblkid-devel libcurl-devel libudev-devel libxml2-devel libuuid-devel ncurses-devel valgrind-devel xfsprogs-devel xmlstarlet yasm nss-devel libibverbs-devel openldap-devel CUnit-devel python3-Cython python3-setuptools python-prettytable lttng-ust-devel expat-devel junit boost-random keyutils-libs-devel openssl-devel libcap-ng-devel python3-sphinx leveldb leveldb-devel snappy snappy-devel lz4 lz4-devel liboath liboath-devel libbabeltrace-devel librabbitmq librabbitmq-devel librdkafka librdkafka-devel libnl3 libnl3-devel rdma-core-devel numactl numactl-devel numactl-libs createrepo openldap-devel rdma-core-devel lz4-devel expat-devel lttng-ust-devel libbabeltrace-devel python3-Cython gperftools-devel bc dnf-plugins-core librabbitmq-devel rpm-build java-1.8.0-openjdk-devel python3-devel openeuler-lsb rpmdevtools cmake gcc-c++ libtool cryptsetup-devel sqlite-devel lua-devel libicu-devel -y

Go to the ceph-16.2.7 directory and run the build script.

cd ceph-16.2.7 && ./do_cmake.sh -DCMAKE_BUILD_TYPE=Release

Go to the build directory and compile ceph-fuse.
1
cd build && make -j ceph-fuse
After the compilation is successful, ceph-fuse is located in the build/bin directory.
1
cd bin

Installing FUSE Dependencies

Perform the following steps on the client node.

Install the libfuse3 module.
1
yum install fuse3 fuse3-devel
Move ceph-fuse in Step 7 in "Preparing the Environment" to the /usr/bin directory and escalate the privilege of ceph-fuse.
1
chmod 755 /usr/bin/ceph-fuse
Install the optimized kernel.
1
yum install kernel-5.10.0-263.0.0.166.oe2203sp4.aarch64
The libfuse library optimization requires modifications to the FUSE kernel module. Those modifications have been supported starting from kernel-5.10.0-263.0.0.166.

Switch the kernel to the version installed in Step 3.

grub2-set-default 'openEuler (5.10.0-263.0.0.166.oe2203sp4.aarch64) 22.03 (LTS-SP4)'

Reboot the server for the new kernel to take effect.
1
reboot

Mounting a Ceph FUSE

Perform the following steps on the client node.

Create a mount point for the file system.
1
mkdir /mnt/cephfs
Run the mount command.
1
ceph-fuse -m 192.168.3.166:6789,192.168.3.167:6789,192.168.3.168:6789 /mnt/cephfs -o use_fastpath
- 192.168.3.166, 192.168.3.167, and 192.168.3.168 are the public IP addresses of the Ceph storage nodes.
- The default port of the MON node is 6789. If use_fastpath is specified after -o, the libfuse optimization module is enabled. Otherwise, the original libfuse module is used.
Confirm that the mounting is successful.
1
df -h
Run the df -h command to check whether the file system is mounted. If the mounting is successful, the file directory in the red box in the following figure is displayed.

Parent topic: General-Purpose Storage