KAE zlib Compression Tuning

Preparing the Environment

Before installing the accelerator engine, you need to apply for and install a license.

License application guide:

https://support.huawei.com/enterprise/en/doc/EDOC1100068118/b9878159

Installation guide:

https://support.huawei.com/enterprise/en/doc/EDOC1100048786/ba20dd15

Download the acceleration engine installation package and developer guide.

https://gitee.com/kunpengcompute/KAEzip/tags

Installing the Acceleration Engine

The developer guide describes how to install and use all modules of the accelerator engine. Select an appropriate installation mode based on the developer guide.

For details, see Installing KAE Using Source Code.

1. Install the acceleration engine according to the developer guide.
2. Install the zlib library.
   1. Download KAEzip and decompress it to the KAEzip-master directory.
   2. Download zlib-1.2.11.tar.gz from the official zlib website and copy it to KAEzip-master/open_source.
   3. Download KAE-kae1.zip to the /home directory, decompress it, and go to the kae_driver directory.

      1 2	unzip KAE-kae1.zip cd KAE-kae1/kae_driver

   4. Create a libwd package and copy it to KAEzip-master/open_source.

      1 2	tar -czvf libwd-1.0.0.tar.gz warpdrive cp libwd-1.0.0.tar.gz KAEzip-master/open_source

   5. Perform the compilation and installation.

      1 2	cd KAEzip-master sh setup.sh install

      The zlib library is installed in /usr/local/kaezip.

3. Replace the zlib software compression algorithm dynamic library.

   1 2 3 4

   cd /usr/local/kaezip/lib cp libz.so.1.2.11 /lib64/ mv /lib64/libz.so.1 /lib64/libz.so.1-bak ln -s /lib64/libz.so.1.2.11 /lib64/libz.so.1

   

   In the cd /usr/local/kaezip/lib command, /usr/local/kaezip/lib indicates the zlib installation path. Change it as required.

If the Ceph cluster is running before the dynamic library is replaced, run the following command on all storage nodes to restart the OSDs for the change to take effect after the dynamic library is replaced:

1	systemctl restart ceph-osd.target

Changing the Default Number of Accelerator Queues

Hardware accelerator queues are resources virtualized by hardware drivers. For example, after the hisi_zip driver is installed, 256 queues are virtualized by default. Each compression task to be sent to the acceleration hardware must apply for a queue in advance. If the number of concurrent tasks exceeds 256, you need to increase the number of accelerator queues. For example, change the number to 512 or 1024.

1. Remove hisi_zip.

   1	rmmod hisi_zip

2. Set the default accelerator queue parameter pf_q_num=512.

   1	vim /etc/modprobe.d/hisi_zip.conf

   1	options hisi_zip uacce_mode=2 pf_q_num=512

3. Load hisi_zip.

   1	modprobe hisi_zip

4. Check the hardware accelerator queue.

   1	cat /sys/class/uacce/hisi_zip-*/attrs/available_instances

   The change is successful if the following information is displayed.

   

5. Check the dynamic library links. If libwd.so.1 is contained in the command output, the operation is successful.

   1	ldd /lib64/libz.so.1

   

Adapting Ceph to the Accelerator

Currently, the mainline Ceph versions allow configuring the zlib compression mode using the configuration file. The released Ceph versions (by version 15.2.3) adopt the zlib compression mode without data headers or trailers. However, the current hardware acceleration library supports only the mode with data headers and trailers. Therefore, the Ceph source code needs to be modified to adapt to the Kunpeng hardware acceleration library. For details about the modification method, see the latest patch that has been incorporated into the mainline versions.

https://github.com/ceph/ceph/pull/34852

The following uses Ceph 14.2.8 as an example to describe how to adapt Ceph to the zlib compression engine.

1. Obtain the source code.

   https://download.ceph.com/tarballs/

   

   After the source package is downloaded, save it to the /home directory on the server.

2. Obtain the patch and save it to the /home directory.

   https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/storage/Patch/

   

3. Go to the /home directory, decompress the source package and go to the directory generated after decompression.

   1	cd /home && tar -zxvf ceph-14.2.8.tar.gz && cd ceph-14.2.8/

4. Apply the patch in the source code directory.

   1	patch -p1 < ../ceph-14.2.8-zlib-compress.patch

5. After modifying the source code, compile Ceph.

   For details about the compilation procedure, see Ceph 14.2.8 Compilation Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).

6. Install Ceph.
7. Modify the ceph.conf file to configure the zlib compression mode.

   1	vi /etc/ceph/ceph.conf

   compressor_zlib_winsize=15

8. Modify the systemd permissions. In Ceph, the RGW process service is managed by systemd. To enable systemd to access hardware acceleration devices, you need to modify the configuration on each RGW node as follows:
   1. Open the configuration file.

      1	vi /usr/lib/systemd/system/ceph-radosgw@.service

      Change PrivateDevices=yes to PrivateDevices=no.

   2. Make the modification take effect.

      1	systemctl daemon-reload

9. Restart the Ceph cluster for the configuration to take effect.

   1	ceph daemon osd.0 config show|grep compressor_zlib_winsize