What's New

1

Updated OmniRuntime features.

• OmniOperator 2.0.0: Added the adaptation layer Gluten 1.3 for Spark, and added the concat_ws, regexp, regexp_replace, trim, and floor expressions for SparkExtension.
• OmniStream 1.1.0: Added the task-level operator rollback mechanism in SQL scenarios, and added checkpoint and restore support for KeyedCoProcess in DataStream scenarios.
• OmniStateStore 1.1.0: Added support for interconnection with the Flink metric framework to implement common metrics, as well as persistent storage of priority queues and KV separated storage.
• Changed the document name from SQL Query Acceleration Feature Documentation to OmniRuntime Feature Documentation.

OmniRuntime Feature Guide

1

Optimized the feature description.

Optimized the feature description for Kunpeng BoostKit for Big Data.

All Kunpeng BoostKit for Big Data documents.

2

Updated OmniRuntime features.

Released the OmniAdvisor 2.0.

OmniRuntime Feature Guide

1

Removed the graph analysis algorithm library documentation.

This feature is no longer iterated or maintained, and related documents are removed.

Graph Analysis Algorithm Feature Guide

1

Updated OmniRuntime features.

• Added the Parquet format to Spark 3.3.1 for OmniOperator to support column-based write.
• Added support for CentOS 7.9 in OmniOperator.
• Added the OmniStateStore feature.
• Added OmniStream Flink Native.

OmniRuntime Feature Guide

1

Removed the TPC-DS Usage Guidance (for Hive & Spark).

TPC-DS has become closed source, and the dataset generation tool is no longer available. As a result, this document has been removed.

-

1

Updated OmniRuntime features and also added some.

• Added support for Spark 3.4.3 and Spark 3.5.2 in OmniOperator.
• Added support for Spark 3.4.3 in OmniMV.
• Added the OmniScheduler feature, which leverages the Yarn load scheduling algorithm to resolve cluster resource unbalance across Hadoop cluster nodes.

OmniRuntime Feature Guide

1

Removed the introduction to hybrid deployments.

Removed the hybrid deployment content from the Technical White Paper and Feature List because the FusionInsight platform has already supported hybrid deployment of x86 and Arm servers.

Kunpeng BoostKit for Big Data Technical White Paper

1

Updated OmniRuntime features.

• Adapted OmniOperator to the Hive engine and added operator and expression details.
• The OmniShuffle Memory Fabric supports a minimum of 1 GB memory. Added support for mistake-proof deployment in External Shuffle Service (ESS) and Remote Shuffle Service (RSS) modes. Added a communication library with higher multi-thread performance in ESS mode.
• Added remote attestation support for OmniShield.
• Added the OmniRuntime public network address statement.
• Added the OmniRuntime open source software notice.

OmniRuntime Feature Guide

2

Added support for enabling the KSAL CRC32 acceleration algorithm in "Kafka Performance Tuning."

Optimized Kafka based on the KSAL software package to improve the throughput of Kafka producers.

Kafka Tuning Guide (CentOS & openEuler)

1

Removed the decoupled storage and compute feature and HBase index optimization feature.

Due to the adjustment of the overall product planning, rarely used features are no longer promoted. Therefore, the decoupled storage and compute feature and the HBase index optimization feature are brought offline.

-

2

Added the OmniRuntime OmniShield feature.

The OmniShield feature runs in the hardware-based TEE and provides data, network, and drive encryption and decryption as well as application-level remote attestation capabilities for Spark. This feature ensures data security throughout the lifecycle of storage, transmission, and computing.

OmniRuntime Feature Guide

3

Added the enablement of the Kunpeng Library LZ4 for Kafka.

The Kunpeng Library is used in Kafka to optimize the LZ4 compression algorithm, improving system performance.

Kafka Tuning Guide (CentOS & openEuler)

4

Added the Doris Instruction Optimization Deployment & Feature Guides.

Doris instruction optimization uses the NEON instruction set to optimize the Bitshuffle running logic on the Arm architecture, further improving the efficiency of LZ4 and Zstandard (Zstd) compression and the data write and query performance of Doris on the Kunpeng server.

Doris Instruction Optimization Deployment & Feature Guides

5

Added the ClickHouse Porting & Deployment & Tuning Guides.

ClickHouse supports linear scalability and features lightweight and high reliability.

ClickHouse Porting & Deployment & Tuning Guides

1

Added openEuler 22.03 LTS support in the big data documentation.

Added openEuler 22.03 LTS support in the big data documentation. For details about porting and deployment, see the related documents in the Kunpeng Community.

Porting Guide

1

Updated OmniRuntime features.

• Added support for Hive 3.1.0 in OmniOperator. The performance of Hive 3.1.0 is improved by 20%.
• Added support for flat-wide table query in the OmniOperator memory optimization. The performance is improved by 30% in the TPC-DS benchmark test of 3 TB data.
• Added support for RSS mode in OmniShuffle. The performance is improved by 10% compared with that of Celeborn.

OmniRuntime Feature Guide

1

Added the page of details about in-depth integration between Kunpeng BoostKit and openEuler.

Integrated the Kunpeng BoostKit acceleration software package into the openEuler image source. This enables out-of-the-box availability and greatly simplifies the installation and usage processes.

-

2

Added the installation guide for integrating big data features with openEuler.

Added the installation, usage, and uninstallation guides for integration between big data features and openEuler.

-

1

Moved the general acceleration feature and HBase acceleration feature to the category of application acceleration.

Moved the general acceleration feature and HBase acceleration feature from the category of basic acceleration to the category of application acceleration.

General Acceleration Feature Guide

HBase Acceleration Feature Guide

2

Updated OmniRuntime features.

• OmniOperator supports Spark 3.3.1.
• OmniShuffle supports optimized NEON instructions, mainly for key read/write processes in the memory.
• OmniAdvisor is enhanced to support Spark applications.

OmniRuntime Feature Guide

1

Updated the Technical White Paper.

Added new features to this document.

Kunpeng BoostKit for Big Data Technical White Paper

2

Updated the document.

Added the OmniAdvisor and OmniHBaseGSI features to version 23.0.0.

OmniRuntime Feature Guide

3

Updated the documents.

Added the algorithm compatible with Spark 3.3.1 to version 23.0.0.

Machine Learning Algorithm Feature Guide

Graph Analysis Algorithm Feature Guide