Enabling the LZ4 Compression Algorithm for Kafka

Purpose

You can enable the LZ4 compression algorithm of the KSAL for Kafka.

Procedure

Replace the native LZ4 code in the lz4-java repository with the LZ4 code of the KSAL. Kafka uses the KafkaLZ4BlockInputStream and KafkaLZ4BlockOutputStream classes to call lz4-java-1.7.1.jar to enable the LZ4 compression algorithm of the KSAL, thereby improving the Kafka performance when it is using the LZ4 compression format.

Software Requirements

Obtaining Software

Table 2 lists the software packages required for enabling the KSAL LZ4 for Kafka.

Configuring the Compilation Environment

1. Install OpenJDK.
   1. Download the installation package.

      1	wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz --no-check-certificate

   2. Decompress the package to the /opt/tools/installed directory.

      1 2 3

      tar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz mkdir -p /opt/tools/installed/ mv jdk8u252-b09 /opt/tools/installed/

   3. Configure environment variables.
      1. Open the profile file.

         1	vi /etc/profile

      2. Press i to enter the insert mode and configure the following content in the file:

         1 2	export JAVA_HOME=/opt/tools/installed/jdk8u252-b09 export PATH=$JAVA_HOME/bin:$PATH

      3. Press Esc, type :wq!, and press Enter to save the file and exit.
   4. Make the environment variables take effect.

      1	source /etc/profile

2. Install Ant.
   1. Download the Ant 1.9.14 installation package.

      1	wget https://archive.apache.org/dist/ant/binaries/apache-ant-1.9.14-bin.zip --no-check-certificate

   2. Decompress the package to the /opt/tools/installed directory.

      1 2	unzip apache-ant-1.9.14-bin.zip mv apache-ant-1.9.14 /opt/tools/installed/

   3. Configure the Ant environment variables.
      1. Open the profile file.

         1	vi /etc/profile

      2. Press i to enter the insert mode and configure the following content in the file:

         1 2	export ANT_HOME=/opt/tools/installed/apache-ant-1.9.14 export PATH=$ANT_HOME/bin:$PATH

      3. Press Esc, type :wq!, and press Enter to save the file and exit.
   4. Make the environment variables take effect.

      1	source /etc/profile

   5. Check whether Ant is installed successfully.

      1	ant -version

      

Compiling the LZ4 Compression Algorithm of the KSAL

1. Download the LZ4 compression algorithm of the KSAL and add a patch to LZ4.
   1. Create an installation directory.

      1	mkdir -p /opt/lz4

   2. Download the LZ4 compression algorithm of the KSAL and decompress it.

      1 2 3

      cd /opt/lz4 wget https://gitee.com/kunpeng_compute/lz4/repository/archive/master.zip --no-check-certificate unzip master.zip

      

   3. Delete the original BoostKit-kzl_1.0.0.zip package.

      1 2	cd /opt/lz4/lz4-master rm -rf BoostKit-kzl_1.0.0.zip

   4. Download the new BoostKit-kzl_1.0.0.zip package from Table 2, upload it to the lz4-master directory, and decompress it.

      1 2	cd /opt/lz4/lz4-master unzip BoostKit-kzl_1.0.0.zip

   5. Decompress the lz4-1.9.3.zip package and add the KZL patch to LZ4.

      1 2 3 4 5 6

      cd /opt/lz4/lz4-master unzip lz4-1.9.3.zip patch -p0 < ./lz4_kzl.patch cp ./kzl.a ./lz4_accelerater.h ./lz4-1.9.3/lib cd lz4-1.9.3/lib ar -x kzl.a

      

2. Download lz4-java and perform compilation.
   1. Download and decompress lz4-java.

      1 2 3

      cd /opt/lz4/ wget https://github.com/lz4/lz4-java/archive/refs/tags/1.7.1.zip --no-check-certificate unzip 1.7.1.zip

   2. Replace the native lz4 directory with the lz4 directory of KZL.

      1 2 3 4

      cd /opt/lz4/lz4-java-1.7.1/src rm -rf lz4 cd /opt/lz4/lz4-master cp -r lz4-1.9.3 /opt/lz4/lz4-java-1.7.1/src/lz4

   3. Modify the build.xml file to compile the .a and .o files into the JAR package.
      1. Go to the /opt/lz4/lz4-java-1.7.1 directory and open the build.xml file.

         1 2	cd /opt/lz4/lz4-java-1.7.1 vim build.xml

      2. Press i to enter the insert mode and change <fileset dir="${src}/lz4/lib" includes="lz4.c, lz4hc.c, xxhash.c" /> in line 214 to the following content:

         1	<fileset dir="${src}/lz4/lib" includes="*.c,*.h, *.a,*.o" />

      3. Press Esc, type :wq!, and press Enter to save the file and exit.
   4. Perform compilation.

      1 2 3

      cd /opt/lz4/lz4-java-1.7.1 ant ivy-bootstrap ant

      

   5. Rename the package as lz4-java-1.7.1.jar.

      1 2	cd dist mv lz4-java-1.6-SNAPSHOT.jar lz4-java-1.7.1.jar

      

Enabling the LZ4 Compression Algorithm of the KSAL

Replace the native package in the libs directory of Kafka with the previously obtained lz4-java-1.7.1.jar package. Restart Kafka.