Performance optimization of BiSheng JDK based on the core big data components Hive and Spark
Supported OSs: CentOS 7.6 and openEuler 20.03 LTS
Supported components: Hive 2.X/3.X and Spark 2.X.
Performance metric: The Hive performance is improved by 2% to 12%, and the Spark performance is improved by 3% to 20%.
BiSheng binary package:
BiSheng JDK software package
Yes
In VM scenarios, BiSheng JDK delivers better performance than OpenJDK of the corresponding version. The actual performance improvement is subject to the VM specifications.
Components: Compatible with Spark 2.3.2, Spark 2.4.5, and Spark 2.4.6. Some of the algorithms support Spark 3.1.1 and Spark 3.3.1, and provide the same interfaces as the open source algorithm library. The other algorithms support Spark 2.X and Spark 3.X technically and need to be adapted as required.
Hardware: Only Kunpeng servers are supported.
Hybrid deployment: The computing queues in a Spark cluster support a mix of Kunpeng servers and servers based on other chip architectures.
Performance metric: Compared with Spark's open source MLlib based on Intel 5318, the machine learning algorithm library based on Kunpeng 5220 improves the computing performance by more than 20% while ensuring the same precision.
JAR files:
Contact Huawei technical support.
Yes
Machine learning algorithms can be used in VM scenarios. The actual performance improvement is subject to the VM specifications.
OmniOperator
Acceleration of native operators based on Spark and Hive. This feature can be used together with OmniShuffle.
Supported components: Spark 3.1.1, Spark 3.3.1, Spark 3.4.3, Spark 3.5.2, and Hive 3.1.0. Other Spark and Hive versions are also supported but will need to be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: OmniOperator improves the computing performance of Spark by 30% and that of Hive by over 20%, according to the 99 TPC-DS SQL benchmark test cases.
JAR files:
Contact Huawei technical support.
Yes
This feature optimizes Spark and Hive compute engine kernels. It is suitable for virtualization scenarios.
OmniShuffle
Shuffle process acceleration based on OCK for Spark. This feature can be used together with OmniOperator.
Supported OSs:
CentOS 7.6, EulerOS 2.0 SP9, and openEuler 20.03 LTS
Components:
NIC driver: Mellanox 5.1-2.4.1.0
JDK 1.8.0_292
GCC 7.3.0
ZooKeeper 3.7.0
Hadoop 3.1.1
Spark 2.4.6 or later
Python 2.7 or later
HiBench 7.1 (recommended)
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric:
In ESS mode:
Typical configuration: 384 GB memory per node, 3+1 servers (three compute nodes and one management node) with two Kunpeng 920 5220 processors per server, at least 10GE network (10GE TCP, 25GE TCP/RDMA, or 100GE TCP/RDMA), and twelve 4 TB SATA drives.
In TeraSort scenario: over 40% higher performance for 1 TB of data
In PageRank (Spark Core) scenario: more than doubled performance for 90 GB of data
In TPC-DS benchmark tests, for 8 TB of data, OmniShuffle improves the Spark performance by 30%, and the combination of OmniShuffle and OmniOperator improves the Spark performance by more than 60%.
In RSS mode:
In TPC-DS benchmark tests, for 3 TB of data, the performance is 10% higher than that of Celeborn.
JAR files:
Contact Huawei technical support.
Yes
This feature depends on network hardware.
The TCP network applies to VM scenarios.
The RDMA network depends on whether RDMA virtualization is supported.
OmniAdvisor 1.0
1.0: Automatic Spark/Hive parameter recommendation using AI
Spark 3.1.1, Spark 3.3.1 and Hive 3.1.0 (only the Hive on Tez mode). Other Spark and Hive versions are also supported but will need to be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: OmniAdvisor improves the Spark performance by 10% according to the 10 TPC-DS SQL benchmark test cases.
JAR files:
Contact Huawei technical support.
Yes
OmniAdvisor applies to VM scenarios.
2.0: Automatic Spark parameter recommendation using AI-driven iterative tuning, expert rule–based tuning, transfer generalization tuning, and operator acceleration modules
Supported OSs: CentOS 7.9 and openEuler 22.03 LTS SP1
Components:
Spark 3.3.1. Other Spark versions are technically supported and can be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: OmniAdvisor improves the Spark performance by 8% using the TPC-DS 3 TB dataset.
JAR files:
Contact Huawei technical support.
Yes
OmniAdvisor applies to VM scenarios.
OmniMV
Intelligent recommendation of materialized views based on Spark and ClickHouse
Spark 3.1.1 and Spark 3.4.3. Other Spark versions are technically supported and can be adapted based on service requirements.
ClickHouse 22.3.6.5. Other Spark versions are technically supported and can be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: OmniMV improves the computing performance of Spark by an average of 30% according to the TPC-DS benchmark test cases, and improves the computing performance of ClickHouse by several times according to Star Schema Benchmark test cases.
JAR files:
Contact Huawei technical support.
Yes
Algorithm models involved in OmniMV are applicable to virtualization scenarios.
OmniScheduler
OmniScheduler leverages the Yarn capacity scheduling policy to allocate containers based on the customized weight sorting of physical and logical resources.
Supported OS: openEuler 22.03 LTS SP3
Components:
Spark 3.1.1, Spark 3.4.3, Hive 3.1.0, and Hadoop 3.3.4. Other Spark versions are technically supported and can be adapted based on service requirements.
Performance metrics: OmniScheduler improves the cluster low-load variance stability by 100% based on TPC-DS benchmark test cases.
JAR file:
Contact Huawei technical support.
Yes
OmniScheduler applies to VM scenarios.
OmniShield
The OmniShield feature runs in the TEE and provides data, network, and drive encryption and decryption as well as application-level remote attestation capabilities for Spark. This feature ensures data security throughout the lifecycle of storage, transmission, and computing.
Supported OS: openEuler 22.03 LTS SP4
Components:
Only 128-bit or 256-bit keys of the AES/GCM/NOPadding algorithm are supported. Only 128-bit keys of the SM4/GCM/NOPadding algorithm are supported.
OmniShield does not provide the KMS service or specify the KMS to be used. Determine what KMS to use by yourself.
Spark 3.1.1 and Spark 3.3.1. Other Spark versions are technically supported and can be adapted based on service requirements.
Performance metric: Based on the 99 TPC-DS benchmark test cases defined by the Big Data Alliance, the performance loss caused by the full-computing link security protection provided by OmniShield does not exceed 20% of the average performance of physical machines.
JAR file:
Contact Huawei technical support.
Yes
OmniShield applies to VM scenarios.
OmniHBaseGSI
HBase global secondary indexes, improving non-rowkey column query efficiency by multiple times
HBase 2.4.14. Other Spark and Hive versions are also supported but will need to be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: OmniHBaseGSI ensure an average latency of less than 30 ms and P99 latency of less than 300 ms in the case of 100 concurrent connections.
This feature is a great choice for big data scenarios where storage and compute are decoupled or coupled at scale.
Supported components: Spark 3.0.0/3.1.1 and Hive 3.1.0 (Tez 0.10.0). Other Spark versions are also supported, but will need to be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metric: According to a TPC-H test, the performance of Spark executing 12 SQL statements is improved by an average of 40% after enabling operator pushdown. According to a TPC-H test, the performance of Hive executing 4 SQL statements is improved by an average of 20% after enabling operator pushdown.
JAR files:
Contact Huawei technical support.
Yes
This feature is suitable for coupled and decoupled storage and compute scenarios that have data locality.
OmniStream
Native code (C/C++) is used based on Flink to implement Flink operators, which improve query performance.
Supported OS: openEuler 22.03 LTS SP4
Components:
Flink 1.16.3. Other Flink versions are technically supported and can be adapted based on service requirements.
Does not support hybrid deployment on Kunpeng servers and servers of other architectures.
Performance metrics: OmniStream improves the computing performance of Flink by an average of over 100% according to Nexmark 22-query test cases. In the WordCount scenario, OmniStream delivers 1.31 times the performance of open source Flink DataStream. In the stateless recomputing scenario, OmniStream achieves 1.6 times the performance of open source Flink DataStream.
JAR file:
Contact Huawei technical support.
Yes
OmniStream applies to VM scenarios.
OmniStateStore
Flink backend plugin that accelerates state storage and improves the overall Flink performance.
Supported OS: openEuler 22.03 LTS SP3
Components:
Flink 1.16.1, 1.16.3, and 1.17.1. Other Flink versions are technically supported and can be adapted based on service requirements.
It can run on both Huawei Kunpeng and general-purpose x86 servers.
Performance metrics: A new state storage technology is introduced to improve the I/O performance of Flink in big data scenarios.
JAR files:
Contact Huawei technical support.
No
It is closely related to hardware networking and is not applicable to virtualization scenarios.
