Documentation
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Installing Dependencies

When installing OmniOperator locally, install the LLVM and jemalloc dependency packages on the management and compute nodes to combine OmniOperator and Spark. In the Spark on Yarn scenario, use the --archives parameter of Spark to simplify the deployment. You can install the dependencies either by downloading precompiled SO files or by compiling source code.

Installing Dependencies (By Downloading Pre-compiled SO Files)

Installing LLVM and jemalloc

The Dependency_library.zip package contains Dependency_library_openeuler.zip, which is applicable to openEuler.

  1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
    1
2
3
    mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
  2. Upload the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) obtained from Obtaining Software Packages to the /opt/omni-operator/ directory and decompress them.
    1
2
    zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
  3. Create an /opt/omni-operator/lib directory, and copy the libLLVM-15.so and libjemalloc.so.2 subpackages in Dependency_library_openeuler to the /opt/omni-operator/lib directory.
    1
2
3
4
5
6
    cd /opt/omni-operator
mkdir lib
rm -rf  /opt/omni-operator/lib/libjemalloc.so.2
rm -rf  /opt/omni-operator/lib/libLLVM-15.so
cp /opt/omni-operator/Dependency_library/Dependency_library_openeuler/libjemalloc.so.2 /opt/omni-operator/lib
cp /opt/omni-operator/Dependency_library/Dependency_library_openeuler/libLLVM-15.so /opt/omni-operator/lib

The /opt/omni-operator and /opt/omni-operator/lib directories can be customized.

Install Dependencies (By Compiling Source Code)

Installing LLVM

  1. Download llvm-project-llvmorg-15.0.4.tar.gz, create an /opt/omni-operator directory on the management and compute nodes as the root directory for installing OmniOperator, go to the directory, and upload the package to /opt/omni-operator.
    1
2
3
4
5
6
    mkdir /opt/omni-operator
cd /opt/omni-operator
tar zxvf llvm-project-llvmorg-15.0.4.tar.gz
mv llvm-project-llvmorg-15.0.4 llvm
cd llvm
mkdir build
  2. Go to the build directory, and compile and install LLVM.
    1
2
3
4
    cd ./build
cmake -DCMAKE_INSTALL_PREFIX=/opt/omni-operator/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=true -DLLVM_ENABLE_PROJECTS="clang" -G "Unix Makefiles" ../llvm
make -j4
make install
  3. Create a lib directory in /opt/omni-operator and copy /opt/omni-operator/llvm/lib/libLLVM-15.so to /opt/omni-operator/lib.
    1
2
    mkdir /opt/omni-operator/lib
cp /opt/omni-operator/llvm/lib/libLLVM-15.so /opt/omni-operator/lib/

The /opt/omni-operator, /opt/omni-operator/llvm, and /opt/omni-operator/lib directories can be customized.

Installing jemalloc

  1. Download jemalloc-5.3.0.tar.gz and upload it to the management and compute nodes.
    1
2
3
    cd /opt/omni-operator/
tar zxvf jemalloc-5.3.0.tar.gz
mv jemalloc-5.3.0 jemalloc

    The /opt/omni-operator/jemalloc directory can be customized.

  2. Go to the jemalloc directory, run the script, and install the generated file.
    1
2
3
    cd jemalloc
./autogen.sh --disable-initial-exec-tls
make -j2
  3. Copy /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 to the /opt/omni-operator/lib directory.
    1
    cp /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 /opt/omni-operator/lib/
Parent topic: Installing the Feature