我要评分
获取效率
正确性
完整性
易理解

Environment Requirements

Before installing OmniData, prepare the hardware and software environments to facilitate subsequent installation operations.

Hardware Requirements

Table 1 lists the hardware requirements for each node in the cluster.

Table 1 Hardware requirements

Item

Description

Processor

Kunpeng 920 5250 processor/New Kunpeng 920 processor model (80 cores)

Memory size

384 GB (12 x 32 GB)

Memory frequency

2666 MHz

Network

  • Ceph environment: 25GE for the service network and 1GE for the management network
  • HDFS environment: 10GE for the service network and 1GE for the management network

Drive

  • System drive: 1 x RAID 0 (1 x 1.2 TB SAS HDD)
  • Data drives (on the management node): 12 x RAID 0 (12 x 4 TB SATA HDD)
  • Data drives (compute node):
    • Ceph environment: 1 x 3.2 TB NVMe
    • HDFS environment: 12 x RAID 0 (12 x 4 TB SATA HDD)
  • Data drives (storage node):
    • Ceph environment: 12 x RAID 0 (12 x 4 TB SATA HDD) 1 x 3.2 TB NVMe
    • HDFS environment: 12 x RAID 0 (12 x 4 TB SATA HDD)

RAID controller card

LSI SAS3508

OS and Software Requirements

Table 2 lists the OS and software requirements for each node in the cluster.

Table 2 OS and software requirements

Item

Version

Description

Management Node

(Server)

Compute Node

(Agent)

Storage Node

(HDFS or Ceph)

OS

  • CentOS 7.6
  • openEuler 20.03 LTS SP1
  • openEuler 22.03 LTS SP1

None

JDK

BiSheng JDK 1.8 (BiSheng JDK 1.8.0_342)

openEuler 22.03 LTS SP1 is incompatible with BiSheng JDK 1.8.0_262, which needs to be replaced with BiSheng JDK 1.8.0_342.

For details about how to install the BiSheng JDK, see BiSheng JDK 8 Installation Guide.

Hadoop

3.2.0

See Hadoop Deployment Guide (CentOS 7.6 & openEuler 20.03).

ZooKeeper

3.6.2

See ZooKeeper Deployment Guide (CentOS 7.6 & openEuler 20.03).

-

-

Hive

3.1.0

See Hive Deployment Guide (CentOS 7.6 & openEuler 20.03).

-

-

Spark

3.1.1

See Installing Spark.

-

-

Ceph

14.2.8

None

-

-

OpenSSL

1.1.1 or later

Link

-

Python

3.10.2 or later

None

-

  • √: indicates that the item is required on the node.
  • -: indicates that the item is not required on the node.
  • If the preceding third-party software has vulnerabilities, fix the vulnerabilities based on official instructions.
  • The preceding component versions may be different from those in the Deployment Guide. The Deployment Guide is for reference only.

Obtaining Software

Table 3 describes the OmniData software packages and how to obtain them.

Table 3 OmniData software packages

Software Name

Package Name

Release Type

Description

How to Obtain

OmniRuntime package

BoostKit-omniruntime_1.7.0.zip

Closed source

OmniRuntime package. Extract the package to obtain the OmniData software package BoostKit-omnidata_1.5.0.zip.

Kunpeng community: Contact Huawei technical support.

NOTE:

Before using the software package, read and agree to Kunpeng BoostKit User License Agreement 2.0.

OmniData plugin for Spark

boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0-aarch64.zip

Open source

Spark extension package for using the OmniData feature.

Link

Third-party dependency package for the OmniData Server.

boostkit-omnidata-server-1.5.0-aarch64-lib.zip

Open source

Third-party dependency package for OmniData.

Obtain this package by following instructions in README.md.

Hive UDF plugin

boostkit-omnidata-hive-udf-loader-1.6.1-1.4.0-aarch64.zip

Open source

Hive UDF plugin package. The Hive UDF plugin is used to execute UDFs written based on the Hive UDF framework.

Obtain boostkit-omnidata-hive-udf-loader-1.6.1.zip and rename it boostkit-omnidata-hive-udf-loader-1.6.1-1.4.0-aarch64.zip. For details, see README.md.

HAF software package

BoostKit-haf_1.4.0.zip

Closed source

HAF software package.

Kunpeng community: Contact Huawei technical support.

NOTE:

Before using the software package, read and agree to Kunpeng BoostKit User License Agreement 2.0.

Dependencies required to access Ceph

hdfs-ceph-3.2.0.jar

Open source

Dependency package required by Hadoop and OmniData to access Ceph.

Link

librgw_jni.so

Open source

Binary file required by Hadoop and OmniData to access Ceph.

Link

Verifying the Software Package Integrity

After downloading a software package from the Kunpeng community, verify the software package to ensure that it is consistent with the original one on the website.

Verify a software package as follows:

  1. Obtain the digital certificate and software.
  2. Obtain the verification tool and guide.
  3. Verify the package integrity by following the procedure described in the OpenPGP Signature Verification Guide obtained from the URL.