Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing SparkExtension

The OmniOperator feature supports the Spark engine. You need to install Spark on the management node and all compute nodes, and configure the SparkExtension dependency for the openEuler OS.

Install the SparkExtension version specific to the used Spark version, for example, SparkExtension 3.1.1 for Spark 3.1.1. You can run the spark-shell --version command to query the Spark version.

Installing OmniOperator requires the Spark extension package and the library files. See Table 3.

  • The boostkit-omniop-spark-3.1.1-1.9.0-aarch64.zip package contains the boostkit-omniop-spark-3.1.1-1.9.0-aarch64-openeuler.zip and boostkit-omniop-spark-3.1.1-1.9.0-aarch64-centos.zip subpackages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace boostkit-omniop-spark-3.1.1-1.9.0-aarch64-openeuler.zip in the following commands with boostkit-omniop-spark-3.1.1-1.9.0-aarch64-centos.zip.
  • The Dependency_library.zip package contains the Dependency_library_openeuler.zip and Dependency_library_centos.zip packages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace Dependency_library_openeuler.zip in the following commands with Dependency_library_centos.zip.

Installing SparkExtension 3.1.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.1.1-1.9.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
    2. Optional: Obtain the Dependency_library packages (Dependency_library.zip, Dependency_library.z01, and Dependency_library.z02) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
3
      zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
\cp -f /opt/omni-operator/Dependency_library/Dependency_library_openeuler/* /opt/omni-operator/lib
    3. Decompress boostkit-omniop-spark-3.1.1-1.9.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-1.9.0-aarch64-openeuler.zip.

      Decompress boostkit-omniop-spark-3.1.1-1.9.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.1.1-1.9.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.1.1-1.9.0-aarch64.jar to /opt/omni-operator/lib.

      Decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.1.1-1.9.0-aarch64.zip
unzip boostkit-omniop-spark-3.1.1-1.9.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.1.1-1.9.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.3.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.3.1-1.9.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
    2. Optional: Obtain the Dependency_library packages (Dependency_library.zip, Dependency_library.z01, and Dependency_library.z02) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
3
      zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
\cp -f /opt/omni-operator/Dependency_library/Dependency_library_openeuler/* /opt/omni-operator/lib
    3. Decompress boostkit-omniop-spark-3.3.1-1.9.0-aarch64.zip to obtain boostkit-omniop-spark-3.3.1-1.9.0-aarch64-openeuler.zip.

      Decompress boostkit-omniop-spark-3.3.1-1.9.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.3.1-1.9.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.3.1-1.9.0-aarch64.jar to /opt/omni-operator/lib.

      Decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.3.1-1.9.0-aarch64.zip
unzip boostkit-omniop-spark-3.3.1-1.9.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.3.1-1.9.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.4.3

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.4.3-1.9.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
    2. Optional: Obtain the Dependency_library packages (Dependency_library.zip, Dependency_library.z01, and Dependency_library.z02) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
3
      zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
\cp -f /opt/omni-operator/Dependency_library/Dependency_library_openeuler/* /opt/omni-operator/lib
    3. Decompress boostkit-omniop-spark-3.4.3-1.9.0-aarch64.zip to obtain boostkit-omniop-spark-3.4.3-1.9.0-aarch64-openeuler.zip.

      Decompress boostkit-omniop-spark-3.4.3-1.9.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.4.3-1.9.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.4.3-1.9.0-aarch64.jar to /opt/omni-operator/lib.

      Decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.4.3-1.9.0-aarch64.zip
unzip boostkit-omniop-spark-3.4.3-1.9.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.4.3-1.9.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.5.2

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.5.2-1.9.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
    2. Optional: Obtain the Dependency_library packages (Dependency_library.zip, Dependency_library.z01, and Dependency_library.z02) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
3
      zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
\cp -f /opt/omni-operator/Dependency_library/Dependency_library_openeuler/* /opt/omni-operator/lib
    3. Decompress boostkit-omniop-spark-3.5.2-1.9.0-aarch64.zip to obtain boostkit-omniop-spark-3.5.2-1.9.0-aarch64-openeuler.zip.

      Decompress boostkit-omniop-spark-3.5.2-1.9.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.5.2-1.9.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.5.2-1.9.0-aarch64.jar to /opt/omni-operator/lib.

      Decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.5.2-1.9.0-aarch64.zip
unzip boostkit-omniop-spark-3.5.2-1.9.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.5.2-1.9.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc
Parent topic: Using on Spark