Related Concepts

Concepts related to Doris instruction optimization.

• The default compression mode for Doris table creation is LZ4.
• In the data compression and storage procedure where LZ4 is involved, Doris first stores data in the page, uses the Bitshuffle algorithm to rearrange the data by bit, and writes the data to bitshuffle_page. Then, Doris uses the LZ4 algorithm to compress the data in bitshuffle_page, to reduce the occupied storage space. This method not only improves data storage and query efficiency, but also lowers storage costs. Zstd compression is similar to LZ4 compression. To use Zstd compression, "compression"="zstd" needs to be added to PROPERTIES for creating data tables.
• The principle of the Bitshuffle algorithm is to divide the input data into multiple small blocks, rearrange each small data block by bit, and make adjacent bits more similar. This increases the data local repetition, facilitating the compression algorithms to easy identify and utilize the repetition, thereby improving the data compression efficiency. Bitshuffle is a lossless compression algorithm that is applicable for processing large-scale scientific datasets, such as images, videos, and meteorological data.
• The NEON instruction set is a 128-bit SIMD instruction set. The instruction set is introduced in ARMv7 and extended in ARMv8 to support vectorized operations. The supported operations include addition, multiplication, comparison, shift, absolute value, maximum and minimum value, as well as saving and loading instructions. Compared with the C language, the NEON instruction set helps achieve higher computing efficiency.