Enabling Gazelle

Introduction

Gazelle is a high-performance protocol stack in user mode. It directly reads and writes NIC packets in user mode based on DPDK and transmits the packets through shared huge page memory. In addition, it uses the lwIP protocol stack, greatly improving the network I/O throughput of applications. Gazelle is mainly used to accelerate the network for databases, such as MySQL and Redis. After Gazelle is used in the high-performance cloud disk optimization solution, the network bottleneck of Ceph can be optimized to improve the cloud disk performance.

Gazelle has the following advantages:

• High performance

  Zero-copy and lock-free packets that can be flexibly scaled out and scheduled adaptively.

• Uniformity

  Compatible with POSIX without modification, and applicable to different types of applications.

Perform the following operations to enable Gazelle.

Installing Gazelle

1. Configure the openEuler Yum repository. Enable the update repository in the openEuler.repo Yum repository file in the /etc/yum.repos.d directory. Change the value of enabled in the [update] repository to 1, as shown in the following:

   [update]
   name=update
   baseurl=http://repo.openeuler.org/openEuler-20.03-LTS-SP1/update/$basearch/
   enabled=1
   gpgcheck=1
   gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS-SP1/OS/$basearch/RPM-GPG-KEY-openEuler

2. Clear the yum cache.

   yum clean all
   yum makecache

3. Use yum to install Gazelle.

   yum install dpdk
   yum install libconfig
   yum install numactl
   yum install libboundscheck
   yum install libpcap
   yum install gazelle

Configuring the Environment

1. Open the lstack.conf file.

   vim /etc/gazelle/lstack.conf

2. Set DPDK parameters.

   dpdk_args=["reinit-ok", "-l", "1", "--log-level", "lib.eal:6", "--log-level", "lib.cryptodev:5", "--log-level", "user1:6", "--match-allocations", "--file-prefix", "spdk_gazelle"]

   

   • Gazelle depends on DPDK. Therefore, the DPDK configuration in the Gazelle configuration file determines the SPDK configuration. When Gazelle is enabled, you need to modify dpdk_args in the preceding configuration file to adjust SPDK parameters.
   • The third parameter indicates the core binding position. The value 1 indicates that the SPDK is bound to CPU 1.
3. Modify or add the following options to the file:

   use_ltran=0
   kni_switch=1
   listen_shadow=1
   mbuf_pool_size = 1024000
   num_cpus="2,3"
   app_bind_numa=0
   main_thread_affinity=1

4. Configure information about the ports to be taken over by Gazelle.

   host_addr="192.168.1.148"
   mask_addr="255.255.255.0"
   gateway_addr="192.168.1.1"
   devices="aa:bb:cc:dd:ee:ff"

   

   Enter the port information as required.

5. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.

Enabling Gazelle

1. Use Gazelle to take over a network port (for example, enp3s0).

   modprobe vfio enable_unsafe_noiommu_mode=1
   modprobe vfio-pci
   ip link set enp3s0 down
   dpdk-devbind -b vfio-pci enp3s0

2. Check whether the network port is successfully taken over.

   dpdk-devbind -s

   

   

   To cancel the takeover, run the following command:

   dpdk-devbind -b hinic 0000:07:00.0

3. Enable Gazelle on the application process.

   Load the Gazelle function library using LD_PRELOAD when running the SPDK startup command.

   LD_PRELOAD=/usr/lib64/liblstack.so" GAZELLE_BIND_PROCNAME=vhost build/bin/vhost -S /var/tmp -m 0x2

   

   • The command for enabling Gazelle is similar to that for enabling TCMalloc. After the combination of the two commands, you can use the following command to start the SPDK:

     TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/tcmalloc LD_PRELOAD="/usr/lib64/libtcmalloc_minimal.so.4 /usr/lib64/liblstack.so" GAZELLE_BIND_PROCNAME=vhost build/bin/vhost -S /var/tmp/ -m 0x02

   • Use the preceding command to start the SPDK during a performance test.
4. Perform core binding when Gazelle is enabled.

   Gazelle running requires separate CPU resources. After the vhost process with Gazelle enabled is started, the core binding mode needs to be adjusted based on your environment.

   Change the range of Ceph client cores for core binding as follows by referring to Binding Threads to CPU Cores:

   taskset -pc 4-12 $pid