Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing SparkExtension

The OmniOperator feature supports the Spark engine. You need to install Spark on the management node and all compute nodes, and configure the SparkExtension dependency for the openEuler OS.

Install the SparkExtension version specific to the used Spark version, for example, SparkExtension 3.1.1 for Spark 3.1.1. You can run the spark-shell --version command to query the Spark version.

Installing OmniOperator requires the Spark extension package and the library files. See Table 3.

  • The boostkit-omniop-spark-3.1.1-2.0.0-aarch64.zip package contains boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip (for NEON) and boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler-sve.zip (for SVE). Select either based on whether the model supports NEON or SVE instructions. The following uses boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip (for NEON) as an example. To install the SVE dependency package on a server that supports SVE instructions, such as Kunpeng 920B, replace boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip with boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler-sve.zip.
  • Select the dependency package based on your OS type. The following uses openEuler 22.03 as an example and the dependency package is Dependency_library_openeuler22.03.zip.

Installing SparkExtension 3.1.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and extract it.

    Download boostkit-omniop-spark-3.1.1-2.0.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
    2. Optional: Obtain the Dependency_library_openeuler22.03.zip package from Obtaining Software Packages and upload it to the /opt/omni-operator directory. Then extract the package content applicable to your OS and copy it to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
      unzip Dependency_library_openeuler22.03.zip
\cp -f /opt/omni-operator/Dependency_library_openeuler22.03/* /opt/omni-operator/lib
    3. Extract boostkit-omniop-spark-3.1.1-2.0.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip.

      Extract boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.1.1-2.0.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.1.1-2.0.0-aarch64.jar to /opt/omni-operator/lib.

      Extract dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.1.1-2.0.0-aarch64.zip
unzip boostkit-omniop-spark-3.1.1-2.0.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.1.1-2.0.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >> ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.3.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and extract it.

    Download boostkit-omniop-spark-3.3.1-2.0.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
    2. Optional: Obtain the Dependency_library_openeuler22.03.zip package from Obtaining Software Packages and upload it to the /opt/omni-operator directory. Then extract the package content applicable to your OS and copy it to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
      unzip Dependency_library_openeuler22.03.zip
\cp -f /opt/omni-operator/Dependency_library_openeuler22.03/* /opt/omni-operator/lib
    3. Extract boostkit-omniop-spark-3.3.1-2.0.0-aarch64.zip to obtain boostkit-omniop-spark-3.3.1-2.0.0-aarch64-openeuler.zip.

      Extract boostkit-omniop-spark-3.3.1-2.0.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.3.1-2.0.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.3.1-2.0.0-aarch64.jar to /opt/omni-operator/lib.

      Extract dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.3.1-2.0.0-aarch64.zip
unzip boostkit-omniop-spark-3.3.1-2.0.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.3.1-2.0.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >> ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.4.3

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and extract it.

    Download boostkit-omniop-spark-3.4.3-2.0.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
    2. Optional: Obtain the Dependency_library_openeuler22.03.zip package from Obtaining Software Packages and upload it to the /opt/omni-operator directory. Then extract the package content applicable to your OS and copy it to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
      unzip Dependency_library_openeuler22.03.zip
\cp -f /opt/omni-operator/Dependency_library_openeuler22.03/* /opt/omni-operator/lib
    3. Extract boostkit-omniop-spark-3.4.3-2.0.0-aarch64.zip to obtain boostkit-omniop-spark-3.4.3-2.0.0-aarch64-openeuler.zip.

      Extract boostkit-omniop-spark-3.4.3-2.0.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.4.3-2.0.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.4.3-2.0.0-aarch64.jar to /opt/omni-operator/lib.

      Extract dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.4.3-2.0.0-aarch64.zip
unzip boostkit-omniop-spark-3.4.3-2.0.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.4.3-2.0.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >> ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.5.2

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and extract it.

    Download boostkit-omniop-spark-3.5.2-2.0.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.

    1. Configure a local Yum repository. The following uses openEuler 22.03 LTS SP1 as an example.
      1
      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/
    2. Install the dependencies.
      1
      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
      mkdir /opt/omni-operator
cd /opt/omni-operator
    2. Optional: Obtain the Dependency_library_openeuler22.03.zip package from Obtaining Software Packages and upload it to the /opt/omni-operator directory. Then extract the package content applicable to your OS and copy it to the /opt/omni-operator/lib directory.
      • If another version of SparkExtension has been installed, skip this step. Check the lib directory in the $OMNI_HOME path. If it contains the .so library and .jar package, this indicates that another version of SparkExtension has been installed. In this document, $OMNI_HOME is /opt/omni-operator.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the copy operation in this step.
      1
2
      unzip Dependency_library_openeuler22.03.zip
\cp -f /opt/omni-operator/Dependency_library_openeuler22.03/* /opt/omni-operator/lib
    3. Extract boostkit-omniop-spark-3.5.2-2.0.0-aarch64.zip to obtain boostkit-omniop-spark-3.5.2-2.0.0-aarch64-openeuler.zip.

      Extract boostkit-omniop-spark-3.5.2-2.0.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.5.2-2.0.0-aarch64.jar and dependencies.tar.gz.

      Move boostkit-omniop-spark-3.5.2-2.0.0-aarch64.jar to /opt/omni-operator/lib.

      Extract dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.5.2-2.0.0-aarch64.zip
unzip boostkit-omniop-spark-3.5.2-2.0.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.5.2-2.0.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    4. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.
      1
2
3
      chmod -R 550 /opt/omni-operator/*
chmod 750 /opt/omni-operator/conf
chmod 640 /opt/omni-operator/conf/omni.conf
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >> ~/.bashrc
source ~/.bashrc