Feature Description

The big data features of OmniRuntime are presented in the form of plugins to improve the performance of data loading, computing, and exchange from end to end.

Data volumes generated from Internet services have been growing much faster than CPUs' computing power. The open-source big data ecosystem is also developing on a fast track. However, diversified computing engines and open source components make it difficult to improve data processing performance throughout the lifecycle. Different big data engines use their own unique tuning policies and technologies to improve performance and efficiency. Some tuning items may be applied across multiple engines, which may cause resource contention and conflicts, reducing overall computing performance.

OmniRuntime consists of a series of features provided by Kunpeng BoostKit for Big Data in terms of application acceleration. It aims to improve the performance of end-to-end data loading, computing, and exchange through plugins, thereby improving the performance of big data analytics.

OmniAdvisor is part of the OmniRuntime feature set.

• OmniAdvisor 1.0: parses parameters of historical Spark and Hive SQL tasks, uses AI algorithms to intelligently tune parameter sampling, and implements end-to-end online parameter tuning for tasks.
  It has been adapted to the following open source components and versions:

  • Spark 3.1.1
  • Spark 3.3.1
  • Hive 3.1.0
  • Tez 0.10.0
• OmniAdvisor 2.0: samples parameters of spark-submit tasks and recommends optimal configurations through AI iterative tuning, expert rule–based tuning, migration generalization tuning, and operator acceleration, enabling end-to-end parameter tuning for Spark tasks.

  It has been adapted to Spark 3.3.1.