Interface Description

Introduction

In modern computing environments, data compression is a key method for improving data transmission efficiency and saving storage space. However, conventional compression methods are usually executed synchronously, which may cause performance bottlenecks when processing large-scale data, especially in high-load scenarios. To better leverage the capabilities of hardware accelerators and improve the system response, Kunpeng BoostKit provides KAELz4 asynchronous compression interfaces.

These interfaces are designed for high-concurrency and large-volume data compression. Concurrent threads are introduced to compress data, which can fully utilize the parallel computing capability of modern multi-core CPUs. With the support of the hardware accelerator KAE, the compression speed is further improved.

These interfaces support both polling and non-polling modes, as well as the block, frame, and lz77_raw3 compression data formats. The block and frame formats are compatible with the standard LZ4 block and frame formats used in the community. The lz77_raw format needs to be converted to a standard block or frame format via the appropriate post-processing interface before being used.

For details about interface combinations of different data formats in different polling modes, see the following sections.

Features

Asynchronous compression: Compression runs asynchronously without blocking the main thread. Results are returned via callback upon completion.
Hardware acceleration: The hardware accelerator boots compression speed, while interfaces enhance compression task efficiency and reduce CPU load.
Multi-thread concurrency: Tasks are distributed evenly through multi-thread intelligent scheduling, further improving the overall performance. In dual-accelerator environments, cores are automatically and evenly distributed across both accelerators, balancing workload and optimizing resource utilization.
LZ4 compatibility: The compressed data generated by interfaces are compatible with the LZ4 format used in the community.
Easy integration: The interfaces can be easily integrated into existing applications. By simply calling interfaces and providing callbacks, you can quickly start asynchronous compression and implement high-performance data compression.

Application Scenarios

High-concurrency and heavy-load scenarios: When handling numerous simultaneous compression tasks, the interfaces support multi-thread execution to prevent single-thread blocking and enhance system responsiveness.
Low-latency scenarios: For systems demanding fast response times, the interfaces maintain smooth operation by avoiding main thread blocking. For example, for real-time data transmission, distributed storage, or streaming media processing, they can effectively prevent frame freezing during data compression.

Prerequisite

The KAELz4 asynchronous compression interfaces are a part of KAELz4 compression of KAE 2.0. To use the interfaces, correctly install KAE 2.0 by referring to Kunpeng Accelerator Engine User Guide.

Error Codes

#define KAE_LZ4_INVAL_PARA 1 # General internal error
#define KAE_LZ4_INIT_FAIL 2 # Failed to initialize resources.
#define KAE_LZ4_COMP_FAIL 3 # Compression failed.
#define KAE_LZ4_RELEASE_FAIL 4 # Failed to release resources.
#define KAE_LZ4_ALLOC_FAIL 5 # Failed to allocate memory resources.
#define KAE_LZ4_SET_FAIL 6 # Internal error
#define KAE_LZ4_HW_TIMEOUT_FAIL 7 # Hardware timeout

Software Computing Scenarios

Software computing is available only when the input data size is less than 64 KB.

Software computing can be switched automatically when the KAE driver is abnormal.
Software computing can be switched automatically when the KAE hardware resources are exhausted.
Software computing can be switched via both universal and polling interfaces.
Soft computing cannot be performed for segmented buffers in SGL mode.

Parent topic: Calling Examples of the KAELz4 Asynchronous Compression Interfaces