Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing SparkExtension

The OmniOperator feature supports the Spark engine. You need to install Spark 3.1.1 or Spark 3.3.1 on the management node and all compute nodes, and configure the SparkExtension dependency (SparkExtension 3.1.1 for Spark 3.1.1 and SparkExtension 3.3.1 for Spark 3.3.1) on the openEuler OS.

  • The boostkit-omniop-spark-3.1.1-1.4.0-aarch64.zip package contains the boostkit-omniop-spark-3.1.1-1.4.0-aarch64-openeuler.zip and boostkit-omniop-spark-3.1.1-1.4.0-aarch64-centos.zip subpackages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace boostkit-omniop-spark-3.1.1-1.4.0-aarch64-openeuler.zip in the following commands with boostkit-omniop-spark-3.1.1-1.4.0-aarch64-centos.zip.
  • The Dependency_library.zip package contains the Dependency_library_openeuler.zip and Dependency_library_centos.zip subpackages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace Dependency_library_openeuler.zip in the following commands with Dependency_library_centos.zip.

Installing SparkExtension 3.1.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.1.1-1.4.0-aarch64.zip from Obtaining Software and upload it to the /opt/omni-operator/ directory on the management node and all compute nodes.

  3. Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension 3.3.1 has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software, upload them to the /opt/omni-operator directory, decompress them to obtain subpackages, and decompress the subpackages applicable to your OS to the /opt/omni-operator/lib directory.

      If SparkExtension 3.3.1 has been installed, skip this step.

      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
unzip -d /opt/omni-operator/lib -o Dependency_library_openeuler.zip
    2. Decompress boostkit-omniop-spark-3.1.1-1.4.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-1.4.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.1.1-1.4.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
      rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.1.1-1.4.0-aarch64.zip
unzip boostkit-omniop-spark-3.1.1-1.4.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Add the following environment variable to the ~/.bashrc file on all nodes:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.3.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.3.1-1.4.0-aarch64.zip from Obtaining Software and upload it to the /opt/omni-operator/ directory on the management node and all compute nodes.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension 3.1.1 has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software, upload them to the /opt/omni-operator directory, decompress them to obtain subpackages, and decompress the subpackages applicable to your OS to the /opt/omni-operator/lib directory.

      If SparkExtension 3.1.1 has been installed, skip this step.

      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
unzip -d /opt/omni-operator/lib -o Dependency_library_openeuler.zip
    2. Decompress boostkit-omniop-spark-3.3.1-1.4.0-aarch64.zip to obtain boostkit-omniop-spark-3.3.1-1.4.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.3.1-1.4.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.3.1-1.4.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.3.1-1.4.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.3.1-1.4.0-aarch64.zip
unzip boostkit-omniop-spark-3.3.1-1.4.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.3.1-1.4.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Optional: Add the following environment variable to the ~/.bashrc file on all nodes:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc
Parent topic: Using on Spark