Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing SparkExtension

The OmniOperator feature supports the Spark engine. You need to install Spark 3.1.1 or Spark 3.3.1 on the management node and all compute nodes, and configure the SparkExtension dependency (SparkExtension 3.1.1 for Spark 3.1.1 and SparkExtension 3.3.1 for Spark 3.3.1) on the openEuler OS. You can run the spark-shell --version command to query the Spark version.

Installing OmniOperator requires the Spark extension package and the library files. See Table 1. This section explains how to install OmniOperator on openEuler. If you want to install it on CentOS, refer to the instructions in the table.

Table 1 Packages required for installing OmniOperator

Software Package

Sub-package

Description

boostkit-omniop-spark-3.1.1-1.7.0-aarch64.zip

boostkit-omniop-spark-3.1.1-1.7.0-aarch64-openeuler.zip

Used for openEuler.

boostkit-omniop-spark-3.1.1-1.7.0-aarch64-centos.zip

Used for CentOS.

To install OmniOperator on CentOS, replace boostkit-omniop-spark-3.1.1-1.7.0-aarch64-openeuler.zip in the following commands with boostkit-omniop-spark-3.1.1-1.7.0-aarch64-centos.zip.

Dependency_library.zip

Dependency_library_openeuler.zip

Used for openEuler.

Dependency_library_centos.zip

Used for CentOS.

To install OmniOperator on CentOS, replace Dependency_library_openeuler.zip in the following commands with Dependency_library_centos.zip.

Installing SparkExtension 3.1.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.1.1-1.7.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension 3.3.1 has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension 3.3.1 has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp /opt/omni-operator/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.1.1-1.7.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-1.7.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.1.1-1.7.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.1.1-1.7.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.1.1-1.7.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
      rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.1.1-1.7.0-aarch64.zip
unzip boostkit-omniop-spark-3.1.1-1.7.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.1.1-1.7.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.3.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.3.1-1.7.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension 3.1.1 has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension 3.1.1 has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in 2.3.2, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp /opt/omni-operator/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.3.1-1.7.0-aarch64.zip to obtain boostkit-omniop-spark-3.3.1-1.7.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.3.1-1.7.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.3.1-1.7.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.3.1-1.7.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.3.1-1.7.0-aarch64.zip
unzip boostkit-omniop-spark-3.3.1-1.7.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.3.1-1.7.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc
Parent topic: Using on Spark