文档
我要评分
获取效率
正确性
完整性
易理解
提交关闭

openEuler

In RocksDB write scenarios, when a large amount of data is written, the background thread needs to compact the data. If the compaction is slow, data writing may be interrupted temporarily. During compaction, a compression algorithm is invoked to compress data. Optimizing the compression algorithm can effectively improve the compaction efficiency and RocksDB write performance.

KAE is a hardware acceleration solution based on the Kunpeng 920 processor. It includes KAE encryption and decryption, KAEZip, KAELz4, and KAEZstd. When RocksDB uses zstd as the compression algorithm during compaction, KAEZstd can be used to accelerate the process.

  1. Apply for and install a license. For details, see Huawei Server iBMC License User Guide.
  2. Obtain KAE-kae2.zip based on Obtaining Software Packages and save it to the /home directory.
  3. Decompress KAE-kae2.zip and go to the KAE source package directory.
    cd /home
unzip KAE-kae2.zip
cd KAE-kae2
  4. Install the dependencies.
    yum install -y numactl-devel pciutils openssl-devel automake m4 perl libtool zlib lzma lz4 make patch
yum install -y kernel-devel-5.10.0-216.0.0.115.oe2203sp4.aarch64

    When installing kernel-devel, run uname -r to query the current kernel version. The kernel-devel version must be the same as the current kernel version.

  5. In the KAE source package directory, perform the following operations to install KAE modules.
    1. Install the kernel driver.
      1. Run the following command:
        sh build.sh driver
      2. Check whether the accelerator engine file system exists.
        ll /sys/class/uacce/

        Information similar to the following is displayed:

        lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_hpre-2 -> ../../devices/pci0000:78/0000:78:00.0/0000:79:00.0/uacce/hisi_hpre-2
lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_hpre-3 -> ../../devices/pci0000:b8/0000:b8:00.0/0000:b9:00.0/uacce/hisi_hpre-3
lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_sec2-0 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.0/uacce/hisi_sec2-0
lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_sec2-1 -> ../../devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/uacce/hisi_sec2-1
lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_zip-4 -> ../../devices/pci0000:74/0000:74:00.0/0000:75:00.0/uacce/hisi_zip-4
lrwxrwxrwx. 1 root root 0 Aug 22 17:14 hisi_zip-5 -> ../../devices/pci0000:b4/0000:b4:00.0/0000:b5:00.0/uacce/hisi_zip-5
      3. Check whether the driver is installed.
        lsmod | grep uacce

        Information similar to the following is displayed:

        uacce                  32768  3 hisi_sec2,hisi_qm,hisi_zip
    2. Install the user-mode driver.
      1. Run the following command:
        sh build.sh uadk
      2. Check whether the installation is successful.
        ll /usr/local/lib/libwd*

        Information similar to the following is displayed:

        -rwxr-xr-x. 1 root root     961 Aug 22 17:23 /usr/local/lib/libwd_comp.la
lrwxrwxrwx. 1 root root      19 Aug 22 17:23 /usr/local/lib/libwd_comp.so -> libwd_comp.so.2.6.0
lrwxrwxrwx. 1 root root      19 Aug 22 17:23 /usr/local/lib/libwd_comp.so.2 -> libwd_comp.so.2.6.0
-rwxr-xr-x. 1 root root  377872 Aug 22 17:23 /usr/local/lib/libwd_comp.so.2.6.0
-rwxr-xr-x. 1 root root     973 Aug 22 17:23 /usr/local/lib/libwd_crypto.la
lrwxrwxrwx. 1 root root      21 Aug 22 17:23 /usr/local/lib/libwd_crypto.so -> libwd_crypto.so.2.6.0
lrwxrwxrwx. 1 root root      21 Aug 22 17:23 /usr/local/lib/libwd_crypto.so.2 -> libwd_crypto.so.2.6.0
-rwxr-xr-x. 1 root root  715616 Aug 22 17:23 /usr/local/lib/libwd_crypto.so.2.6.0
-rwxr-xr-x. 1 root root     907 Aug 22 17:23 /usr/local/lib/libwd.la
lrwxrwxrwx. 1 root root      14 Aug 22 17:23 /usr/local/lib/libwd.so -> libwd.so.2.6.0
lrwxrwxrwx. 1 root root      14 Aug 22 17:23 /usr/local/lib/libwd.so.2 -> libwd.so.2.6.0
-rwxr-xr-x. 1 root root 1342080 Aug 22 17:23 /usr/local/lib/libwd.so.2.6.0
    3. Compile and install KAEZstd.
      1. Run the following command to install KAEZstd:
        sh build.sh zstd
      2. Check whether the installation is successful.
        ll /usr/local/kaezstd/
        The installation is successful if information similar to the following is displayed:
        drwxr-xr-x 2 root root 4096 Jun 11 22:29 bin
drwxr-xr-x 2 root root 4096 Jun 11 22:29 include
drwxr-xr-x 3 root root 4096 Jun 11 22:29 lib
drwxr-xr-x 3 root root 4096 Jun 11 22:29 share
      3. View generated files.
        ll /usr/local/kaezstd/bin/
        Information similar to the following is displayed:
        lrwxrwxrwx 1 root root      4 Jun 11 22:29 unzstd -> zstd
-rwxr-xr-x 1 root root 986192 Jun 11 22:29 zstd
lrwxrwxrwx 1 root root      4 Jun 11 22:29 zstdcat -> zstd
-rwxr-xr-x 1 root root   3869 Jun 11 22:29 zstdgrep
-rwxr-xr-x 1 root root     30 Jun 11 22:29 zstdless
lrwxrwxrwx 1 root root      4 Jun 11 22:29 zstdmt -> zstd
  6. Check the number of hardware device queues to determine whether the program has invoked KAE to accelerate the zstd algorithm. For example, run the following command to check the number of queues corresponding to the hisi_zip driver module. By default, the number is 256.
    watch -n 0.2 cat /sys/class/uacce/hisi_zip-*/available_instances
    Command output:
    Every 0.2s: cat /sys/class/uacce/hisi_zip-0/available_instances/sys/class/uacce/hisi_zip...  localhost.localdomain:
256
256
256
256

    Start another window and perform 7. If the number of queues changes, hardware-based KAE compression acceleration is successfully enabled.

    Every 0.2s: cat /sys/class/uacce/hisi_zip-0/available_instances/sys/class/uacce/hisi_zip...  localhost.localdomain:
256
254
256
256
Parent topic: Installing KAEZstd