Installing the Spark Engine

The following operations must be performed on the management node and all compute nodes.

1. Install the Spark engine. For details, see OS and Software Requirements.
2. Install the SparkExtension dependency of the openEuler OS.

   Configure the local yum source for each OS image and run the following commands to install the dependencies:

   yum install lz4-devel.aarch64 -y
   yum install zstd-devel.aarch64 -y
   yum install snappy-devel.aarch64 -y
   yum install protobuf-c-devel.aarch64 protobuf-lite-devel.aarch64 -y
   yum install boost-devel.aarch64 -y
   yum install cyrus-sasl-devel.aarch64 -y
   yum install jsoncpp-devel.aarch64 -y
   yum install openssl-devel.aarch64 -y
   yum install libatomic.aarch64 -y

3. Configure SparkExtension.
   1. Obtain the ORC and Protobuf software packages from Obtaining Software. Decompress them to obtain liborc.so and libprotobuf.so.24 and upload the files to the /opt/omni-operator/lib directory. Then set the permission on the files to 550.

      chmod 550 /opt/omni-operator/lib/liborc.so
      chmod 550 /opt/omni-operator/lib/libprotobuf.so.24

   2. Decompress boostkit-omniop-spark-3.1.1-1.2.0-aarch64.zip to obtain boostkit-omniop-spark-3.1.1-1.2.0-aarch64.jar, copy boostkit-omniop-spark-3.1.1-1.2.0-aarch64.jar to the /opt/omni-operator/lib directory, and set the permission on the software package to 550.

      chmod 550 /opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.2.0-aarch64.jar

   3. Decompress the dependencies.tar.gz package extracted from boostkit-omniop-spark-3.1.1-1.2.0-aarch64.zip and copy the dependencies folder to the /opt/omni-operator/lib directory. Set the permission on the software package to 550.

      chmod -R 550 /opt/omni-operator/lib/dependencies/

4. Add the following environment variable to the ~/.bashrc file on all nodes:

   export OMNI_HOME=/opt/omni-operator