Documentation
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Installing SparkExtension

The OmniOperator feature supports the Spark engine. You need to install Spark on the management node and all compute nodes, and configure the SparkExtension dependency for the openEuler OS. Install the SparkExtension version specific to the used Spark version, for example, SparkExtension 3.1.1 for Spark 3.1.1. You can run the spark-shell --version command to query the Spark version.

Installing OmniOperator requires the Spark extension package and the library files. See Table 3.

Installing SparkExtension 3.1.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.1.1-1.8.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension of another version has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension of another version has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp opt/omni-operator/Dependency_library/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.1.1-1.8.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-1.8.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.1.1-1.8.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.1.1-1.8.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.1.1-1.8.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
      rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.1.1-1.8.0-aarch64.zip
unzip boostkit-omniop-spark-3.1.1-1.8.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.1.1-1.8.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.3.1

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.3.1-1.8.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension of another version has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension of another version has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp /opt/omni-operator/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.3.1-1.8.0-aarch64.zip to obtain boostkit-omniop-spark-3.3.1-1.8.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.3.1-1.8.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.3.1-1.8.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.3.1-1.8.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.3.1-1.8.0-aarch64.zip
unzip boostkit-omniop-spark-3.3.1-1.8.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.3.1-1.8.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.4.3

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.4.3-1.8.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension of another version has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension of another version has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp /opt/omni-operator/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.4.3-1.8.0-aarch64.zip to obtain boostkit-omniop-spark-3.4.3-1.8.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.4.3-1.8.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.4.3-1.8.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.4.3-1.8.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.4.3-1.8.0-aarch64.zip
unzip boostkit-omniop-spark-3.4.3-1.8.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.4.3-1.8.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc

Installing SparkExtension 3.5.2

  1. Install Spark. For details, see OS and Software Requirements.
  2. Download the SparkExtension plugin package and decompress it.

    Download boostkit-omniop-spark-3.5.2-1.8.0-aarch64.zip from Obtaining Software Packages and upload it to the /opt/omni-operator/ directory on the management node.

  3. Optional: Install the SparkExtension dependency of openEuler.

    Configure the local yum source for each OS image and run the following commands to install the dependencies:

    If SparkExtension of another version has been installed, skip this step.

    1
    yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y
  4. Configure SparkExtension.
    1. Optional: Obtain the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) from Obtaining Software Packages and upload them to the /opt/omni-operator directory. Then decompress the subpackages applicable to your OS and copy them to the /opt/omni-operator/lib directory.
      • If SparkExtension of another version has been installed, skip this step.
      • If you have copied libLLVM-15.so and libjemalloc.so.2 to /opt/omni-operator/lib in Installing Dependencies, skip the operation of copying the two SO files.
      1
2
3
4
5
      cd /opt/omni-operator
rm -rf Dependency_library_*.zip
zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
cp /opt/omni-operator/Dependency_library_openeuler/* /opt/omni-operator/lib
    2. Decompress boostkit-omniop-spark-3.5.2-1.8.0-aarch64.zip to obtain boostkit-omniop-spark-3.5.2-1.8.0-aarch64-openeuler.zip. Then decompress boostkit-omniop-spark-3.5.2-1.8.0-aarch64-openeuler.zip to obtain boostkit-omniop-spark-3.5.2-1.8.0-aarch64.jar and dependencies.tar.gz. Move boostkit-omniop-spark-3.5.2-1.8.0-aarch64.jar to the /opt/omni-operator/lib directory and decompress dependencies.tar.gz to the /opt/omni-operator/lib directory.
      1
2
3
4
5
6
7
      cd /opt/omni-operator
rm -rf dependencies.tar.gz
unzip boostkit-omniop-spark-3.5.2-1.8.0-aarch64.zip
unzip boostkit-omniop-spark-3.5.2-1.8.0-aarch64-openeuler.zip
mv boostkit-omniop-spark-3.5.2-1.8.0-aarch64.jar ./lib
tar -zxvf dependencies.tar.gz -C ./lib
rm -f *.zip
    3. Change the permission on the installation package to 550.
      1
      chmod -R 550 /opt/omni-operator/lib/*
  5. Optional: Add the following environment variable to the ~/.bashrc file on the management node:
    1
2
    echo "export OMNI_HOME=/opt/omni-operator" >>  ~/.bashrc
source ~/.bashrc
Parent topic: Using on Spark