Enabling the CRC32 Acceleration Algorithm for Kafka

Purpose

This section describes how to enable the Kunpeng BoostKit SDS KSAL CRC32 acceleration algorithm for Kafka 2.5.0. You can also refer to this section for other Kafka versions. The acceleration algorithm accelerates the CRC32 algorithm processing performance during Kafka network transmission and improves the end-to-end performance.

Procedure

Apply the function code patch to open source Kafka (deeply customize the Kafka check patch for closed-source version and perform modifications), and modify the Kafka compilation script. Kafka components use CRC32C to determine the factory class of CRC32C. In its static code block, a branch is added for Arm-based machines, and JNI is used to call the Kunpeng Storage Acceleration Library (KSAL) to enable CRC32C.

Software Requirements

**Table 1** Software requirements
Item	Supported Version
Java	OpenJDK
OS	openEuler 22.03 LTS SP3
Kafka	1.0.0-3.8.0
Gradle	6.1.1

Obtaining Software

Table 2 lists the software packages required for enabling the Kunpeng BoostKit SDS KSAL CRC32 for Kafka.

**Table 2** Software packages required for enabling the KSAL CRC32 for Kafka
Name	Software Package Name	Release Type	Description	How to Obtain
Kunpeng Storage Acceleration Library (KSAL)	BoostKit-KSAL_1.6.0.zip	Closed source	A Huawei-developed storage algorithm library that uses algorithms optimized based on Kunpeng instead of mainstream open source algorithms to improve storage performance. It contains the Erasure Code (EC), Cyclic Redundancy Check 16 T10 Data Integrity Field (CRC16 T10DIF), and Cyclic Redundancy Check 32 (CRC32C) algorithms.	Link
Kafka patch of KSAL	ksal-crc32-code.patch	Open source	Patch that integrates KSAL, to be used during Kafka compilation	Link

Configuring the Compilation Environment

Install OpenJDK.

Download the installation package.

          
               wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz --no-check-certificate

Decompress the package to the /opt/tools/installed directory.

          
               tar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz
mkdir -p /opt/tools/installed/
mv jdk8u252-b09 /opt/tools/installed/

Configure environment variables.

Open the profile file.

            
                 vi /etc/profile

Press i to enter the insert mode and configure the following content in the file:

            
                 export JAVA_HOME=/opt/tools/installed/jdk8u252-b09
export PATH=$JAVA_HOME/bin:$PATH

Press Esc, type :wq!, and press Enter to save the file and exit.

Make the environment variables take effect.

          
               source /etc/profile

Check whether OpenJDK is installed successfully.

If the JDK version is "1.8.0_252" as displayed, the installation is successful.

           
                java -version

Install Gradle.

Download the Gradle installation package.

          
               wget https://services.gradle.org/distributions/gradle-6.1.1-bin.zip --no-check-certificate

Decompress the package to the /opt/tools/installed directory.

          
               unzip gradle-6.1.1-bin.zip
mv gradle-6.1.1 /opt/tools/installed/

Configure environment variables for Gradle.

Open the profile file.

            
                 vi /etc/profile

Press i to enter the insert mode and configure the following content in the file:

            
                 export GRADLE_HOME=/opt/tools/installed/gradle-6.1.1
export PATH=$GRADLE_HOME/bin:$PATH

Press Esc, type :wq!, and press Enter to save the file and exit.

Make the environment variables take effect.

          
               source /etc/profile

Check whether Gradle is installed successfully.
1

gradle -version

Install KSAL.

Download the KSAL binary package.

          
               wget https://kunpeng-repo.obs.cn-north-4.myhuaweicloud.com/Kunpeng%20BoostKit/Kunpeng%20BoostKit%2024.0.RC2/BoostKit-KSAL_1.6.0.zip --no-check-certificate

Install KSAL.

          
               unzip BoostKit-KSAL_1.6.0.zip
rpm -ivh libksal-release-1.6.0.oe1.aarch64.rpm

Check whether the installation is successful.

If the KSAL SO packages exist in the directory, the installation is successful. See the following figure.

           
                ll /lib64/*ksal*

Compiling Kafka

Download the patch that is to enable the KSAL CRC32 for Kafka.

Create an installation directory.

          
               mkdir -p /opt/crc32

Download the CRC32 acceleration algorithm of the KSAL and decompress it.

          
               cd /opt/crc32 
wget https://gitee.com/kunpengcompute/boostkit-bigdata/repository/archive/main.zip --no-check-certificate 
unzip main.zip

Download Kafka, use the patch, and compile Kafka.

Download and decompress Kafka.

          
               cd /opt/crc32 
wget https://github.com/apache/kafka/archive/refs/tags/2.5.0.zip --no-check-certificate 
unzip 2.5.0.zip

Use the patch that integrates KSAL.

          
               cd kafka-2.5.0
git apply /opt/crc32/boostkit-bigdata-main/kafka/ksal-crc32-code.patch

Modify the Gradle compilation script.

Use the patch to modify build.gradle for open source Kafka 2.5.0.

           
                git apply /opt/crc32/boostkit-bigdata-main/kafka/ksal-crc32-build-code.patch

For other Kafka versions, modify the compilation script file as follows:

Open build.gradle.

                
                     vi build.gradle

Locate the corresponding lines to be modified using /project(':clients') {.

Press i to enter the insert mode and add the following content to the project(':clients') module in the file:

                
                     task compileCrcJNI(type:Exec) {
    def javaHome = System.getenv('JAVA_HOME')
    commandLine 'gcc', '-fPIC', '-I', javaHome + '/include', '-I', javaHome + '/include/linux',
            '-shared', '-o', 'src/main/resources/libNativeCrc32C.so',
            'src/main/native/org/apache/kafka/common/utils/KsalCrc32C.c', '-lksal'
} 
compileJava.dependsOn 'compileCrcJNI'

Press Esc, type :wq!, and press Enter to save the file and exit.

In Kafka 2.7.2, an error is reported indicating that the code is incompatible when the native Scala version is compiled. You need to modify the Scala in the gradle.properties file to an earlier version, for example, 2.12.10 (recommended).

Perform compilation.

          
               gradle -g /$USER_HOME/gradleRepository releaseTarGz -info

Obtain the Kafka binary package kafka_2.12-2.5.0.tgz.

          
               ll core/build/distributions/

Enabling the CRC32 Acceleration Algorithm

Decompress the Kafka binary package.

        
             tar -zxvf core/build/distributions/kafka_2.12-2.5.0.tgz -C /usr/local

Start ZooKeeper and Kafka.

        
             cd /usr/local/kafka_2.12-2.5.0
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
bin/kafka-server-start.sh -daemon config/server.properties

Create a topic.

        
             bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 70 --topic test

Send a message.

        
             bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
randomtest

This script is provided by Kafka and is used to send messages to a specified Kafka topic.

bootstrap-server indicates the IP address and port number of the Kafka broker.
topic indicates the topic to which data is written.

After performing this operation, you can enter any message and press Enter to send the message. To abort, press Ctrl + C to exit.

Stop sending the message and check Kafka logs. If KSAL-related logs are generated, the deployment takes effect.
1

cat logs/server.log | grep KSAL

Parent topic: Kafka Performance Tuning