README

Project Introduction

The Intelligent Storage Acceleration Library (ISA-L) is a collection of low-level functions optimized for storage applications. ISA-L includes:

• Erasure coding (EC): fast block-level Reed-Solomon type erasure coding, supporting arbitrary encoding/decoding matrices in GF(2^8).
• Cyclic redundancy check (CRC): fast CRC implementation, supporting six different polynomials: iscsi32, ieee32, t10dif, ecma64, iso64, jones64, and rocksoft64.
• RAID: XOR and P+Q parity calculations and operations for common RAID implementations.
• Compression: fast deflate-compatible data compression.
• Decompression: fast inflate-compatible data decompression.
• igzip: a gzip-like command-line application accelerated with ISA-L.

The EC and CRC iscsi32 (CRC32C) algorithms are optimized for Kunpeng servers. The specific optimizations are as follows:

CRC32C

Based on the instruction set features supported by the CPU and to fully utilize all logical computing units:

• For new Kunpeng 920 processor models, the original 3-way CRC32 parallel computation is extended to 6-way parallel computation.
• On Kunpeng 950 processors, a new mixed implementation combining SVPMULL instructions with scalar CRC32 instructions has been added.

In addition, optimizations for scenarios where data hit rates in the cache are low have been made based on different user needs:

• For cache-unfriendly scenarios, more compatible cache prefetch optimization instructions have been added to the original implementation.

EC

The generation of parity blocks uses a table lookup method. Based on the characteristics of the table, the calculation of the first parity block can be replaced with an XOR operation, thereby improving calculation efficiency. Currently, the calculation is optimized only for common ratios x+1, x+2, and x+3.

Test Results Before and After CRC and EC Optimizations

• CRC32C test

  Block Size	New Kunpeng 920 Processor Model - Before	New Kunpeng 920 Processor Model - After	Kunpeng 950 Processor - Before	Kunpeng 950 Processor - After
  4K	33304 MB/s	37633 MB/s	27050 MB/s	45428 MB/s
  8K	33531 MB/s	37890 MB/s	26815 MB/s	57314 MB/s

• EC test

  Block Size	Data Block Quantity	Parity Block Quantity	Encoding/Decoding	Kunpeng 950 Processor - Before	Kunpeng 950 Processor - After
  11904	10	1	Encoding 1	15278 MB/s	79672 MB/s
  11904	10	1	Decoding 1	15267 MB/s	77571 MB/s
  21824	4	2	Encoding 1	19851 MB/s	33196 MB/s
  21824	4	2	Decoding 1	8918 MB/s	95945 MB/s
  11904	8	3	Encoding 1	13152 MB/s	20981 MB/s
  11904	8	3	Decoding 1	13559 MB/s	81694 MB/s

For details about the test procedure, see User Guide.

The block sizes in the EC test are generated by default by the test tool. The generation rule is: 128K / (Number of data blocks + Number of parity blocks).

Directory Structure

├── docs                                    # Project documentation
│   ├── LICENSE                             # Document license
│   └── en                                  # Document directory
│       ├── installation_guide.md           # Installation guide
│       ├── release_notes.md                # Release notes
│       └── user_guide.md                   # User Guide
├── arm-for-ec-crc32c-optimization.patch    # Patch file for EC and CRC32C optimization on Kunpeng servers
├── LICENSE                                 # Software license
└── README.md                               # Project introduction

User Guide

For details about the performance test procedure of ISA-L CRC and EC, see User Guide.

Release Notes

For details about the version updates of ISA-L, see Release Notes.

Installation Guide

For details about how to compile and install ISA-L, see Installation Guide.

Disclaimer

To Users of This Project

• This project is intended solely for debugging and development. You are responsible for any risks and should carefully review the following information:

  • Data processing and deletion: Users are responsible for managing and deleting any data generated while using this tool. You are advised to promptly delete any related data after use to prevent information leaks.
  • Data confidentiality and transmission: Users understand and agree not to share or transmit any data generated by this tool. Neither the tool nor its developers are responsible for any information leaks, data breaches, or other negative consequences.
  • User input security: Users are responsible for the security of any commands they enter and for any risks or losses resulting from improper input. The tool and its developers are not liable for issues caused by incorrect command usage.

• Disclaimer scope: This disclaimer applies to all individuals and entities using this tool. By using the tool, you acknowledge and accept this statement and assume all risks and responsibilities arising from its use. If you do not agree, please stop using the tool immediately.

• Before using this tool, please read and understand the preceding disclaimer. If you have any questions, contact the developer.

To Data Owners

If you do not want your model or dataset to be mentioned in this project, or if you wish to update its description, please submit an issue on GitCode. We will delete or update your description according to your request. Thank you for your understanding and contribution to this project.

License

The code of this project is licensed under BSD-3-Clause. For details, see LICENSE.

The documents of this project are licensed under CC-BY 4.0. For details, see LICENSE.

Contribution Statement

We welcome your contributions to the community. If you have any questions/suggestions or want to provide feedback on feature requirements and bug reports, you can submit issues. For details, see the contribution guideline. You are also welcome to share insights in Discussions. Thank you for your support.