Introduction

Feature Overview

The development of emerging technologies such as 5G and AI has accelerated the generation and flow of data. According to Huawei's Global Industry Vision (GIV) report, the global data volume will reach 180 ZB by 2025. Diversified services are bringing unprecedented growth of data, and data is becoming more and more important. As the data volume increases, applications have higher requirements on storage system performance. Emerging applications have increasingly high performance requirements. How to improve storage system performance to meet service requirements has become a great challenge.

The KSAL is developed by Huawei. It contains the EC , CRC16 T10DIF, and CRC32C algorithms.

Based on the Huawei-developed vectorized EC encoding and decoding solution, the EC algorithm replaces the high-order finite field GF(2^w) multiplication required in the erasure coding process with binary matrix multiplication through isomorphism mapping, so as to use exclusive or (XOR) instead of complex finite field multiplication that is implemented through table lookup. In addition, the EC algorithm uses an encoding orchestration algorithm to reuse intermediate results in the parity block calculation process, which reduces XOR operands and accelerates encoding by working with Kunpeng vectorized instructions. Compared with open source EC, the KSAL EC algorithm has better performance, delivering 2x encoding performance than mainstream open source EC.

CRC16 T10DIF and CRC32C use a modulo algorithm for large numbers and Kunpeng vectorized instructions to accelerate encoding. Compared with open source algorithms, the 4 KB verification performance of CRC16 T10DIF is 30% higher, and the 4 KB verification performance of CRC32C is 130% higher.

This document describes how to obtain, install, deploy, verify, and use the KSAL and how to enable the KSAL EC algorithm in Ceph.