zlib v1.2.12 Optimization Patch

Project Introduction

zlib is a general-purpose DEFLATE compression library that provides support for zlib, gzip, and raw DEFLATE data formats. Using the open-source zlib v1.2.12 tag as a baseline, this deliverable provides standalone optimization patch files. The optimization is primarily targeted at the Linux AArch64 platform, particularly Kunpeng processors. It maintains compatibility with the open-source zlib external APIs, main ABI versions, and compressed data formats. By utilizing Arm NEON/SVE, CRC instructions, wide-word read/write, Huffman fast tables, and compression match algorithm adjustments, it reduces instruction and memory access overhead in the hot paths of compression and decompression.

Optimization Principles

Compression Path

1. On AArch64, CRC32 instructions are used to calculate 4-byte hashes, reducing the overhead of per-byte rolling hashes; other architectures retain the generic implementation.
2. The longest-match comparison is modified to perform batch XOR operations using machine words, and ctz is used to quickly locate the first differing byte, reducing per-byte comparisons and branches.
3. Compression levels 1 to 3 use a streamlined fast matching path; levels 4 to 6 use a new deflate_medium delayed matching flow; higher levels adjust hash chain search parameters to control search costs.
4. On AArch64, sliding window hash table updates are processed in batches using NEON or SVE saturating subtraction.
5. The minimum optimized match length is set to 4 bytes, increasing the utilization of CRC hashes and wide-word comparisons. This strategy may cause slight differences in the compressed output byte sequences and compression ratios compared to the original version, but the output remains standard DEFLATE data.

Bitstream Encoding

1. The deflate bit buffer is expanded from 16 bits to 64 bits, reducing the frequency of writing short bit segments.
2. Length and distance codes use precomputed combined tables, merging codewords, extra bits, and base values for processing.
3. On little-endian platforms, 16-, 32-, and 64-bit batch writes are used for the pending buffer, lowering byte-level write overhead.

Decompression Path

1. When the input and output buffers are large enough, a wider Huffman fast decoding table is constructed, merging symbols, code lengths, base values, and extra bits into the table entries.
2. The fast loop uses 64-bit little-endian input reads, reducing the frequency of replenishing the bit buffer.
3. The NEON/SVE path uses vectorized chunk copies to handle normal and overlapping match replication, improving throughput during the inflate output stage.
4. The path automatically falls back to the original decoding table and state machine when input or output space is insufficient, ensuring correctness in streaming and small-buffer scenarios.

Adler-32 and Platform Selection

1. AArch64 CPUs that support dot-product use NEON DOT instructions to compute Adler-32.
2. The SVE Adler-32, sliding hash, and decompression copy paths are enabled when the --sve configuration is used.
3. The default AArch64 build runs check_cpu_feature_dot.sh to detect DOT instructions; if DOT is not supported, the Armv8 CRC and NEON decompression paths are still used.

Environment Requirements

• Recommended environment: Linux AArch64, Kunpeng processor, and GCC or compatible compilers
• The default mode requires Armv8 CRC and NEON; DOT instructions are detected at build time.
• The SVE mode requires both the compiler and the running CPU to support SVE.
• tar, make, and git or patch are required.
• Native compilation on the target AArch64 machine is recommended. The automatic architecture selection of configure depends on the local uname results, making it unsuitable as a generic cross-compilation configuration.
• The build hardening and linking parameters in the patch files are primarily targeted at Linux. The verification scope of this deliverable does not cover MacOS, Windows, and other Unix platforms.

Quick Start

Automatic Patch Application

Steps for Automatic Patch Application

Obtain the deliverable and run the following commands in the deliverable directory to execute the build.sh script:

chmod +x build.sh
./build.sh

The build.sh script performs the following operations and outputs information about the source build directory and installation directory upon completion:

1. Creates an independent build directory under tmp/.
2. Decompresses the zlib source package zlib-1.2.12.tar.gz.
3. Verifies and applies the zlib-optimizs-v1.2.12.patch file.
4. Executes configure and parallel compilation.
5. Executes make test by default.
6. Installs to the install/ subdirectory of the current build directory by default.

SVE Build

Execute the following command to perform an SVE build only when both the target processor and the compiler are confirmed to support SVE. SVE is a compile-time selection and does not include runtime automatic degradation. The SVE version cannot be deployed on processors that do not support SVE.

./build.sh --sve

Common Build Parameters

JOBS=16 ./build.sh
RUN_TESTS=0 ./build.sh
RUN_INSTALL=0 ./build.sh
BUILD_DIR=tmp/zlib-opt-build ./build.sh
INSTALL_PREFIX=/opt/zlib-1.2.12-opt ./build.sh

The directory specified by BUILD_DIR must not exist beforehand. If it is not specified, the script creates a randomly named directory under tmp/.

Manual Patch Application

The patch file must be applied to clean zlib v1.2.12 source code. Other zlib versions or already modified source code may cause conflicts; forcing the application is not recommended.

Steps for Manual Patch Application

mkdir -p tmp/manual
tar -xzf zlib-1.2.12.tar.gz -C tmp/manual --strip-components=1
cd tmp/manual

git apply --check --whitespace=nowarn ../../zlib-optimizs-v1.2.12.patch
git apply --whitespace=nowarn ../../zlib-optimizs-v1.2.12.patch

./configure --prefix="$PWD/install"
make -j"$(nproc)"
make test
make install

If Git is not installed, you can use the following command to apply the patch:

patch -p1 < ../../zlib-optimizs-v1.2.12.patch

Using the Optimized Library

After installation, the header files and libraries are located in the installation directory output upon script completion.

Dynamically linked programs can select the optimized library via the installation path or LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/path/to/install/lib:${LD_LIBRARY_PATH:-}
ldd /path/to/application | grep libz

Alternatively, it can be explicitly specified when building upper-level software:

CPPFLAGS="-I/path/to/install/include" \
LDFLAGS="-L/path/to/install/lib" \
./configure

The usage of zlib interfaces remains identical to the open-source v1.2.12. Before replacing the system libz.so, you should verify dependent applications in an isolated environment. Directly overwriting the library provided by the OS is not recommended.

Verification Suggestions

1. Retain make test to confirm that the example and minigzip tests for both static and shared libraries pass.
2. Use real-world service data to test compression throughput, decompression throughput, compression ratio, and CPU usage respectively.
3. Decompress data generated by the optimized version using the original zlib, and decompress data generated by the original zlib using the optimized version, to verify two-way format compatibility.
4. Use ldd to confirm that the service application actually loads the optimized libz.so.1.
5. Build and deploy separately on SVE and non-SVE machines to avoid copying SVE binaries to environments that do not support SVE.

Constraints

• Performance gains depend on data distribution, compression levels, input block sizes, compilers, and CPU microarchitectures; service stress test results shall prevail.
• Compression match heuristics and chain length parameters have been adjusted, so the compression ratio may differ slightly from the original v1.2.12.
• This deliverable is based on the historical version v1.2.12. Before deploying it to production environments, you need to evaluate upstream subsequent fixes and version maintenance strategies in conjunction with your service security requirements.
• The optimized source code contains zlib, Arm/Chromium BSD, MIT, and Huawei copyright and license notices declared in its files; the original notices must be retained during distribution.