Installing Gluten

The OmniOperator feature supports the Spark engine. You need to install Spark on the management node and all compute nodes, and configure the Gluten dependency for the openEuler OS.

1. Install Spark. For details, see OS and Software Requirements.
   

   Gluten supports only Spark 3.3.1. You can run the spark-shell --version command to check the current Spark version.

2. Download the Gluten plugin package and extract it.

   Obtain Boostkit-omniruntime-gluten-1.0.0.zip and Dependency_library_Gluten.zip from Obtaining Software Packages and upload them to the /opt/omni-operator/ directory on the management node.

3. Install the Gluten dependency of openEuler.
   1. Configure a local yum repository. The following uses openEuler 22.03 LTS SP1 as an example.

      dnf config-manager --add-repo https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS-SP1/extras/aarch64/

   2. Install the dependencies.

      yum install lz4-devel zstd-devel snappy-devel protobuf-c-devel protobuf-lite-devel boost-devel cyrus-sasl-devel jsoncpp-devel openssl-devel libatomic -y

4. Configure Gluten.
   1. Extract Boostkit-omniruntime-gluten-1.0.0.zip and Dependency_library_Gluten.zip to /opt/omni-operator/lib.

      cd /opt/omni-operator
      unzip BoostKit-omniruntime-gluten-1.0.0.zip
      unzip Dependency_library_Gluten.zip
      unzip BoostKit-omniruntime-omnioperator-2.0.0.zip
      tar -zxvf boostkit-omniop-operator-2.0.0-aarch64-openeuler-sve.tar.gz
      mkdir lib
      mv libboundscheck.so libspark_columnar_plugin.so gluten-omni-bundle-spark3.3_2.12-openEuler_22.03_aarch_64-1.3.0.jar lib
      mv Dependency_library_Gluten/lib* lib/
      mv boostkit-omniop-operator-2.0.0-aarch64/libboostkit-omniop-* lib/

   2. Change the permission on the program file in the software package to 550, on the configuration file directory to 750, and on the configuration file to 640.

      chmod -R 550 /opt/omni-operator/*
      chmod 750 /opt/omni-operator/conf
      chmod 640 /opt/omni-operator/conf/omni.conf

5. Add the following environment variable to the ~/.bashrc file on the management node:

   1 2	echo "export OMNI_HOME=/opt/omni-operator" >> ~/.bashrc source ~/.bashrc