Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing Dependencies

When installing OmniOperator locally, install the LLVM and jemalloc dependency packages on the management node to combine OmniOperator and Spark.

In the Spark on Yarn scenario, use the --archives parameter of Spark to simplify the deployment. You can install the dependencies either by downloading precompiled SO files or by compiling source code. The precompiled SO file can be downloaded and installed quickly, and is suitable for most scenarios. In contrast, compiling and installing from source code is slower, but may be required for certain compliance scenarios.

Installing Dependencies (By Downloading Precompiled SO Files for SparkExtension)

Installing LLVM and jemalloc

  • Select the dependency package based on your OS type. The following uses openEuler 22.03 as an example and the dependency package is Dependency_library_openeuler22.03.zip.
  • The /opt/omni-operator and /opt/omni-operator/lib directories can be customized.
  1. Create an /opt/omni-operator/ directory on the management node as the root directory for installing OmniOperator. Then go to the directory.
    1
2
    mkdir /opt/omni-operator
cd /opt/omni-operator
  2. Upload the Dependency_library_openeuler22.03.zip package obtained from Obtaining Software Packages to the /opt/omni-operator/ directory and extract it.
    1
    unzip Dependency_library_openeuler22.03.zip
  3. Create an /opt/omni-operator/lib directory, and copy the libLLVM-15.so and libjemalloc.so.2 subpackages in Dependency_library_openeuler to the /opt/omni-operator/lib directory.

    If LLVM and jemalloc have been installed in the environment, delete the existing libLLVM-15.so and libjemalloc.so.2 files before running the copy command.

    1
2
    rm -rf /opt/omni-operator/lib/libjemalloc.so.2
rm -rf /opt/omni-operator/lib/libLLVM-15.so

    1
2
3
4
    cd /opt/omni-operator
mkdir lib
cp /opt/omni-operator/Dependency_library_openeuler22.03/libjemalloc.so.2 /opt/omni-operator/lib
cp /opt/omni-operator/Dependency_library_openeuler22.03/libLLVM-15.so /opt/omni-operator/lib

Installing Dependencies (By Downloading Precompiled SO Files for Gluten)

Installing LLVM and jemalloc

  • The precompiled SO files of the ABSL library are not provided for Gluten. Therefore, you need to manually compile and install the SO files.
  • The /opt/omni-operator and /opt/omni-operator/lib directories can be customized.
  1. Create an /opt/omni-operator/ directory on the management node as the root directory for installing OmniOperator. Then go to the directory.
    1
2
3
    mkdir /opt/omni-operator
cd /opt/omni-operator
rm Dependency_library_Gluten.zip -rf
  2. Upload the Dependency_library_Gluten.zip package obtained from Obtaining Software Packages to the /opt/omni-operator/ directory and extract it.
    1
    unzip Dependency_library_Gluten.zip
  3. Create an /opt/omni-operator/lib directory, and copy the libLLVM-15.so and libjemalloc.so.2 subpackages in Dependency_library_Gluten to the /opt/omni-operator/lib directory.

    If LLVM and jemalloc have been installed in the environment, delete the existing libLLVM-15.so and libjemalloc.so.2 files before running the copy command.

    1
2
    rm -rf /opt/omni-operator/lib/libjemalloc.so.2
rm -rf /opt/omni-operator/lib/libLLVM-15.so

    1
2
3
4
    cd /opt/omni-operator
mkdir lib
cp /opt/omni-operator/Dependency_library_Gluten/libjemalloc.so.2 /opt/omni-operator/lib
cp /opt/omni-operator/Dependency_library_Gluten/libLLVM-15.so /opt/omni-operator/lib
  4. Compile and install ABSL. For details, see 1 to 3 in the procedure for installing ABSL.

Installing Dependencies (By Compiling and Installing Source Code for SparkExtension and Gluten)

Installing LLVM

The /opt/omni-operator, /opt/omni-operator/llvm, and /opt/omni-operator/lib directories can be customized.

  1. Download llvm-project-llvmorg-15.0.4.tar.gz, create an /opt/omni-operator directory on the management node as the root directory for installing OmniOperator, go to the directory, and upload the package to /opt/omni-operator.
    1
2
3
4
5
6
    mkdir /opt/omni-operator
cd /opt/omni-operator
tar zxvf llvm-project-llvmorg-15.0.4.tar.gz
mv llvm-project-llvmorg-15.0.4 llvm
cd llvm
mkdir build
  2. Go to the build directory, and compile and install LLVM.
    1
2
3
4
    cd ./build
cmake -DCMAKE_INSTALL_PREFIX=/opt/omni-operator/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=true -DLLVM_ENABLE_PROJECTS="clang" -G "Unix Makefiles" ../llvm
make -j4
make install
  3. Create a lib directory in /opt/omni-operator and copy /opt/omni-operator/llvm/lib/libLLVM-15.so to /opt/omni-operator/lib.
    1
2
    mkdir /opt/omni-operator/lib
cp /opt/omni-operator/llvm/lib/libLLVM-15.so /opt/omni-operator/lib/

Installing jemalloc

  1. Download jemalloc-5.3.0.tar.gz and upload it to the management node.
    1
2
3
    cd /opt/omni-operator/
tar zxvf jemalloc-5.3.0.tar.gz
mv jemalloc-5.3.0 jemalloc

    The /opt/omni-operator/jemalloc directory can be customized.

  2. Go to the jemalloc directory, run the script, and install the generated file.
    1
2
3
    cd jemalloc
./autogen.sh --disable-initial-exec-tls
make -j2
  3. Copy /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 to the /opt/omni-operator/lib directory.
    1
    cp /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 /opt/omni-operator/lib/

Installing ABSL (required only when ABSL is enabled for Gluten)

  1. Download the ABSL source code to the management node.
    git clone https://github.com/abseil/abseil-cpp.git
cd abseil-cpp/
git checkout tags/20250127.0
  2. Compile the ABSL source code.
    mkdir build && cd build
cmake ..   -DCMAKE_CXX_STANDARD=17   -DCMAKE_CXX_STANDARD_REQUIRED=ON   -DABSL_PROPAGATE_CXX_STD=ON -DBUILD_SHARED_LIBS=ON
make -j32
make install
  3. Copy the compiled ABSL library to /opt/omni-operator/lib.
    cp /usr/local/lib64/libabsl_* /opt/omni-operator/lib
Parent topic: Installing the Feature