Deploying the Spark Engine

The following operations must be performed on the management node and all compute nodes.

1. Deploy the Spark engine. For details, see Software Requirements.
2. Install the SparkExtension dependencies for CentOS and openEuler. For details about the differences, see the NOTE information.

   Configure the local yum source for each OS image and run the following commands to install the dependencies:

   yum install lz4-devel.aarch64 -y
   yum install zstd-devel.aarch64 -y
   yum install snappy-devel.aarch64 -y
   yum install protobuf-c-devel.aarch64 protobuf-lite-devel.aarch64 -y
   yum install boost-devel.aarch64 -y
   yum install cyrus-sasl-devel.aarch64 -y
   yum install jsoncpp-devel.aarch64 -y
   yum install openssl-devel.aarch64 -y
   yum install libatomic.aarch64 -y

   

   zstd is missing in the yum source of the CentOS image. You need to manually compile and install zstd. After the compilation and installation are complete, copy the compilation target SO file of zstd to the /usr/lib64 directory. For details, see 5.4.

3. Configure SparkExtension.
   1. Obtain the ORC and Protobuf software packages from Software Requirements. Decompress them to obtain liborc.so and libprotobuf.so.26. Then upload them to the /opt/omni-operator/lib directory and set the permission on the software package to 550.

      chmod 550 /opt/omni-operator/lib/liborc.so
      chmod 550 /opt/omni-operator/lib/libprotobuf.so.26

   2. Copy the boostkit-omniop-spark-3.1.1-1.1.0-aarch64.jar file decompressed from boostkit-omniop-spark-3.1.1-1.0.0-aarch64.zip to /opt/omni-operator/lib. Set the permission on the software package to 550.

      chmod 550 /opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.1.0-aarch64.jar

   3. Decompress the dependencies.tar.gz file decompressed from boostkit-omniop-spark-3.1.1-1.0.0-aarch64.zip and copy the dependencies folder to the /opt/omni-operator/lib directory. Set the permission on the software package to 550.

      chmod -R 550 /opt/omni-operator/lib/dependencies/

   4. Obtain the OpenSSL software package from Software Requirements. Decompress them to obtain libcrypto.so.1.1 and libssl.so.1.1. Then upload them to the /opt/omni-operator/lib directory. Set the permission on the software package to 550.

      chmod 550 /opt/omni-operator/lib/libcrypto.so.1.1
      chmod 550 /opt/omni-operator/lib/libssl.so.1.1

      

      Perform this step only for CentOS. openEuler has libcrypto.so and libssl.so of this version inside.

   5. Add the following environment variables to the ~/.bashrc file on all nodes. (If the OmniOperator lib path of an earlier version exists, delete it first.)

      export LD_LIBRARY_PATH=/opt/omni-operator/lib:$LD_LIBRARY_PATH
      export OMNI_CONNECTED_ENGINE=Spark