Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Software Requirements

Category

Supported Version

Description

Management Node

Server

Compute Node

Agent

Storage Node

(HDFS or Ceph)

OS

OmniData Server is compatible with CentOS 7.6, openEuler 20.03 LTS SP1, and openEuler 22.03 LTS.

-

√

√

√

JDK

BiSheng JDK 1.8 (BiSheng JDK 1.8.0_262 recommended)

openEuler 22.03 LTS is incompatible with BiSheng JDK 1.8.0_262, which needs to be replaced with BiSheng JDK 1.8.0_342.

√

√

√

Hadoop

Hadoop 3.2.0

See Hadoop Deployment Guide (CentOS 7.6 & openEuler 20.03).

√

√

√

ZooKeeper

ZooKeeper 3.6.2

See ZooKeeper Deployment Guide (CentOS 7.6 & openEuler 20.03).

-

√

-

Hive

Hive 3.1.0

See Hive Deployment Guide (CentOS 7.6 & openEuler 20.03).

√

-

-

Ceph

Ceph 14.2.8 binary package

Ceph 14.2.8 source code

-

-

-

√

Spark

Spark 3.0.0/Spark 3.1.1

See Spark Deployment Guide (CentOS 7.6 & openEuler 20.03).

√

-

-

OpenSSL

OpenSSL 1.1.1 or later

-

-

√

√

Python

Python 3.9.2 or later

-

-

√

√
  • You can install the big data engine Spark or Hive as required.
  • The storage service can be HDFS or Ceph.
  • The HAF service consumes random numbers on your system. To prevent program blocking, you must enable the Haveged service.
    1. Install Haveged. Skip this step if Haveged has been installed.
      yum install -y haveged
    2. Enable the Haveged service and configure it to start as the system boots.
      systemctl start haveged
systemctl enable haveged.service
    3. Confirm the service is enabled.
      service haveged status
  • If the OpenSSL version is earlier than 1.1.1, the certificate fails to be issued. Upgrade OpenSSL to 1.1.1 or later.
    1. Check the Open SSL version. If it is earlier than 1.1.1, upgrade the version.
      openssl version
    2. Obtain the software package.
      wget https://www.openssl.org/source/openssl-1.1.1n.tar.gz
    3. Decompress, compile, and install the package.
      tar -zxvf openssl-1.1.1n.tar.gz
cd openssl-1.1.1n
sudo ./config --prefix=/usr/local/openssl
sudo make -j 16
sudo make install
sudo mv /usr/bin/openssl /usr/bin/openssl.bak
sudo ln -sf /usr/local/openssl/bin/openssl /usr/bin/openssl
    4. Open the file and add /usr/local/openssl/lib to the end of the file.
      sudo vim /etc/ld.so.conf
    5. Make the setting to take effect.
      sudo ldconfig -v

Obtaining Software

Table 1 lists the software packages required for OmniData and how to obtain them.

Table 1 OmniData software packages

Item

Package Name

Release Type

Download URL

Description

OmniData binary package

BoostKit-omnidata_1.4.0.zip

Closed source

Contact Huawei technical support.

OmniData installation package.

OmniData plugin for Spark

boostkit-omnidata-spark-sql_2.12-3.0.0-1.4.0-aarch64.zip

Open source

Link

Spark engine extension package for using the OmniData function.

boostkit-omnidata-spark-sql_2.12-3.1.1-1.4.0-aarch64.zip

Open source

Link

OmniData plugin for Hive

boostkit-omnidata-hive-exec-3.1.0-1.4.0.zip

Open source

Link

Hive engine extension package for using the OmniData function.

Third-party dependency package of OmniData Server

boostkit-omnidata-server-1.4.0-aarch64-lib.zip

Open source

Compile this package. For details, see README.md.

Third-party dependency package for OmniData.

UDF plugin for Hive

boostkit-omnidata-hive-udf-loader-1.6.1-1.4.0-aarch64.zip

Open source

Compile boostkit-omnidata-hive-udf-loader-1.6.1.zip and rename it to boostkit-omnidata-hive-udf-loader-1.6.1-1.4.0-aarch64.zip. For details, see README.md.

Hive UDF plugin package.

The Hive UDF plugin is used to execute UDFs compiled based on the Hive UDF framework.

HAF binary package

BoostKit-haf_1.3.0.zip

Closed source

Contact Huawei technical support.

HAF installation package.

Dependencies required to access Ceph

hdfs-ceph-3.2.0.jar

Open source

Link

Dependency package required by Hadoop and OmniData to access Ceph.

librgw_jni.so

Open source

Link

Binary file required by Hadoop and OmniData to access Ceph.

Verifying Software Package Integrity

After downloading the closed-source binary packages, verify the software packages to ensure that they are consistent with the original software packages on the corresponding websites.

Verify the software package packages as follows:

  1. Obtain the digital certificate and software.
  2. Obtain the verification tool and method from the following link:
  3. Verify the package integrity by following the procedure described in the OpenPGP Signature Verification Guide obtained from the URL.
Parent topic: Environment Requirements