Product Introduction

• You agree to comply with national laws, regulations, and public ethics when using the Kunpeng BoostKit OmniData operator pushdown feature. You shall not use this feature to engage in any activities that violate the law, infringe on the rights and interests of others, disrupt social order, undermine social stability, or engage in any activities that endanger or attempt to endanger the computer system and network security.

• You agree that Huawei shall assume no liability for any indirect, incidental, special, or any form of punitive damages, or any loss of profits, revenue, data, or data use.
• You acknowledge and agree that you need to download and integrate the open source and third-party software on which the software package of this feature depends. Huawei does not assume any responsibility for the software vulnerabilities and security issues.

The OmniData feature is suitable for big data compute-storage decoupling or large scale convergence scenarios where a large number of compute nodes read data from remote nodes. In such scenarios, a large amount of raw data is transmitted from storage nodes to compute nodes over the network for processing. The proportion of valid data is low, greatly wasting network bandwidth.

OmniData performs the following functions:

• Pushes down the Filter, Aggregation, and Limit operators from compute nodes to storage nodes for computing and transmits the computing results to compute nodes over the network, reducing the amount of data transmitted over the network and improving the computing performance of Spark.

• Connects to the homogeneous acceleration framework (HAF) to replace the server/client interface of the original gRPC communication pushdown framework, and implements pushdown using annotations.

• Pushes operators down to Ceph or HDFS storage nodes for processing.