Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Error Reported When Ceph Uses KAEzip to Compress a File Larger Than 4 MB

Symptom

When a file larger than 4 MB is used to test the zlib hardware computing compression rate of Ceph object storage, the error message "Compression error: compress unused input" is displayed.

Key Process and Cause Analysis

Key process:

  • Triggering scenarios

    In the following scenarios, the last data record may not be compressed:

    • The Deflate interface of zlib is directly used for compression.
    • A file larger than 4 MB is compressed in block mode.
    • KAEzip exits in advance without compressing the last data record when the output buffer is small and data records need to be compressed in batches. Z_OK is returned and strm.vaild_in is not processed completely.
    • The application software does not determine that the valid data to be compressed is 0 (strm.vaild_in).
  • Affected versions

    For Kunpeng servers that use KAEzip hardware acceleration, this problem exists in KAEzip 1.3.10 and earlier versions.

  • Example of the problem code
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    int err = deflateInit(&strm, level);
if (err != Z_OK) return err;
strm.avail_out = 0;
strm.avail_in = sourceLen; //The size is greater than 4 MB.
strm.next_in = (Bytef*)source;
int flush = Z_FINISH;

char* szBuf = new char[CEPH_PAGE_SIZE];
do {
memset(szBuf, 0, CEPH_PAGE_SIZE); //CEPH_PAGE_SIZE=64*1024
strm.next_out = (Bytef*)szBuf;
strm.avail_out = CEPH_PAGE_SIZE;

ret = deflate(&strm, flush);    /* no bad return value */
if (ret == Z_STREAM_ERROR) {
deflateEnd(&strm);
return -1;
}
have = CEPH_PAGE_SIZE - strm.avail_out;

memcpy(dest + outnum, szBuf, have);
outnum += have;
} while (strm.avail_out == 0);

//When the compression exits in the preceding scenarios, ret = Z_OK and strm.avail_in! = 0.
*destLen = outnum;

if (strm.avail_in != 0) {
deflateEnd(&strm);
return -1; //Returns an error message.
}

    If you do not follow the zlib Usage Example and do not make the judgment shown in the following figure, the compression of the last data record is not complete.

Root cause:

KAEzip has a cache for compressed output. When the external space is insufficient, KAEzip caches the data that is not transmitted. When the compression interface is called next time, the data in the cache is obtained first. If this is the last time that the compression interface is called, KAEzip exits after obtaining the cache without performing compression.

Conclusion and Solution

This problem exists in KAE 1.3.10 and earlier versions and has been resolved in version 1.3.11 (released on May 20, 2021). Upgrade the KAEzip software package to 1.3.11 to solve this problem.

  • If the KAEzip software is installed using an RPM or DEB package, perform the following steps:
    1. Stop the current services and processes to release the KAEzip resources.
    2. Upgrade the KAEzip software package by referring to "Installation, Upgrade, and Uninstallation" > "Upgrading the KAE Software" in Kunpeng Accelerator Engine Developer Guide (KAEzip).
  • If the KAEzip software is installed using source code, perform the following steps:
    1. Stop the current services and processes to release the KAEzip resources.
    2. Uninstall the original KAEzip by referring to "Installation, Upgrade, and Uninstallation" > "Uninstalling the KAE Software" > "Uninstalling the KAE Software Installed Using Source Code" in Kunpeng Accelerator Engine Developer Guide (KAEzip).
    3. Install KAEzip of the required version by referring to "Installation, Upgrade, and Uninstallation" > "Installing the KAE Software" > "Installing the KAE Software Package Using Source Code" in Kunpeng Accelerator Engine Developer Guide (KAEzip).
Parent topic: Software