Rate This Document
Findability
Accuracy
Completeness
Readability

Environment Requirements

Before installing OmniOperator, prepare the hardware and software environments to facilitate subsequent installation operations.

Hardware Requirements

Table 1 lists the hardware requirements for each node in the cluster.

Table 1 Hardware requirements

Item

Management/Compute/Storage Node

Processor

Kunpeng 920 series

NOTICE:

OmniOperator can be enabled for Gluten only on servers that support the SVE instruction set. You can run cat /proc/cpuinfo | grep sve | head -n 1 to check whether the SVE instruction set is supported. If any command output is displayed, the SVE instruction set is supported.

Memory size

384 GB (12 x 32 GB)

Memory frequency

2666 MHz

Network

  • Service network: 10GE
  • Management network: 1GE

Drive

  • System drive: 1 x RAID 0 (1 x 1.2 TB SAS HDD)
  • Data drive: 12 x RAID 0 (12 x 8 TB SATA HDD)

RAID controller card

LSI SAS3508

OS and Software Requirements

Table 2 lists the OS and software requirements for each node in the cluster.

Table 2 OS and software requirements

Item

Version

Description

Management Node (Server)

Compute/Storage Node

OS

  • CentOS 7.9
  • openEuler 20.03 LTS SP1
  • openEuler 22.03 LTS SP1

Later patch versions such as openEuler 20.03 LTS SP3 and openEuler 22.03 LTS SP3 are also supported.

JDK

BiSheng JDK 1.8 (BiSheng JDK 1.8.0_342)

openEuler 22.03 LTS SP1 is incompatible with BiSheng JDK 1.8.0_262, which needs to be replaced with BiSheng JDK 1.8.0_342.

For details about how to install the BiSheng JDK, see BiSheng JDK 8 Installation Guide.

Hadoop

3.2.0

See Hadoop Deployment Guide (CentOS 7.6 & openEuler 20.03).

Spark

See Spark Deployment Guide (CentOS 7.6 & openEuler 20.03).

-

Hive

3.1.0

See Hive Deployment Guide (CentOS 7.6 & openEuler 20.03).

-

Python

3.10.2 or later

None

  • √: indicates that the item is required on the node.
  • -: indicates that the item is not required on the node.
  • If the preceding third-party software has vulnerabilities, fix the vulnerabilities based on official instructions.
  • The preceding component versions may be different from those in the Deployment Guide. The Deployment Guide is for reference only.

Obtaining Software Packages

Table 3 lists the software packages required for installing the OmniOperator feature and explains how to obtain them. In subsequent operations, install the required installation packages based on the operation guide.

Use on Spark:

  • SparkExtension requires installing the software packages numbered 1, 2 (select the SparkExtension version according to the Spark version), and 5.
  • Gluten requires installing the software package numbered 4.

Use on Hive:

  • HiveExtension requires installing the software packages numbered 1, 3, and 5.
Table 3 OmniOperator software packages

No.

Software Name

Package Name

Release Type

Description

How to Obtain

1

OmniRuntime package

BoostKit-omniruntime_1.9.0.zip

Closed source

OmniRuntime package (BoostKit-omniruntime_1.9.0.zip). Extract the package to obtain the OmniOperator software package BoostKit-omniop_2.0.0.zip.

Kunpeng community

NOTE:

Before using the software package, read and agree to Kunpeng BoostKit User License Agreement 2.0.

2

SparkExtension

boostkit-omniop-spark-3.1.1-2.0.0-aarch64.zip

Open source

Spark extension package for the OmniOperator computing base.

Link

boostkit-omniop-spark-3.3.1-2.0.0-aarch64.zip

Open source

Spark extension package for the OmniOperator computing base.

Link

boostkit-omniop-spark-3.4.3-2.0.0-aarch64.zip

Open source

Spark extension package for the OmniOperator computing base.

Link

boostkit-omniop-spark-3.5.2-2.0.0-aarch64.zip

Open source

Spark extension package for the OmniOperator computing base.

Link

3

HiveExtension

boostkit-omniop-hive-3.1.0-2.0.0-aarch64.zip

Open source

Hive extension package for the OmniOperator computing base.

Link

4

Gluten

Boostkit-omniruntime-gluten-1.0.0.zip

Open source

OmniOperator software installation package (adapted to Gluten)

Link

Dependency_library_Gluten.zip

Library file on which Gluten depends.

Link

5

Dependency_library

Dependency_library_centos.zip

Dependency_library_openeuler20.03.zip

Dependency_library_openeuler22.03.zip

Open source

Library file on which OmniOperator depends. Select the dependency package that matches your OS type.

CentOS dependencies

openEuler 20.03 dependencies

openEuler 22.03 dependencies

Verifying the Software Package Integrity

After downloading a software package from the Kunpeng community, verify the software package to ensure that it is consistent with the original one on the website.

Verify a software package as follows:

  1. Obtain the digital certificate and software.
  2. Obtain the verification tool and guide.
  3. Verify the package integrity by following the procedure described in the OpenPGP Signature Verification Guide obtained from the URL.