文档
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Installing Dependencies

When installing OmniOperator locally, install the LLVM and jemalloc dependency packages on the management node to combine OmniOperator and Spark. In the Spark on Yarn scenario, use the --archives parameter of Spark to simplify the deployment. You can install the dependencies either by downloading pre-compiled SO files or by compiling source code.

Installing Dependencies (By Downloading Pre-compiled SO Files)

Installing LLVM and jemalloc

The Dependency_library.zip package contains the Dependency_library_openeuler.zip and Dependency_library_centos.zip subpackages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace Dependency_library_openeuler.zip in the following commands with Dependency_library_centos.zip.

  1. Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
    1
2
3
    mkdir /opt/omni-operator
cd /opt/omni-operator
rm -rf Dependency_library_*.zip
  2. Upload the Dependency_library packages (Dependency_library.zip and Dependency_library.z01) obtained from Obtaining Software Packages to the /opt/omni-operator/ directory and decompress them.
    1
2
    zip -F Dependency_library.zip --out Dependency_library_complete.zip
unzip Dependency_library_complete.zip
  3. Create an /opt/omni-operator/lib directory, and copy the libLLVM-15.so and libjemalloc.so.2 subpackages in Dependency_library_openeuler to the /opt/omni-operator/lib directory.
    1
2
3
4
5
6
    cd /opt/omni-operator
mkdir lib
rm -rf  /opt/omni-operator/lib/libjemalloc.so.2
rm -rf  /opt/omni-operator/lib/libLLVM-15.so
cp /opt/omni-operator/Dependency_library_openeuler/libjemalloc.so.2 /opt/omni-operator/lib
cp /opt/omni-operator/Dependency_library_openeuler/libLLVM-15.so /opt/omni-operator/lib

The /opt/omni-operator and /opt/omni-operator/lib directories can be customized.

Install Dependencies (By Compiling Source Code)

Installing LLVM

  1. Download llvm-project-llvmorg-15.0.4.tar.gz, create an /opt/omni-operator directory on the management and compute nodes as the root directory for installing OmniOperator, go to the directory, and upload the package to /opt/omni-operator.
    1
2
3
4
5
6
    mkdir /opt/omni-operator
cd /opt/omni-operator
tar zxvf llvm-project-llvmorg-15.0.4.tar.gz
mv llvm-project-llvmorg-15.0.4 llvm
cd llvm
mkdir build
  2. Go to the build directory, and compile and install LLVM.
    1
2
3
4
    cd ./build
cmake -DCMAKE_INSTALL_PREFIX=/opt/omni-operator/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=true -DLLVM_ENABLE_PROJECTS="clang" -G "Unix Makefiles" ../llvm
make -j4
make install
  3. Create a lib directory in /opt/omni-operator and copy /opt/omni-operator/llvm/lib/libLLVM-15.so to /opt/omni-operator/lib.
    1
2
    mkdir /opt/omni-operator/lib
cp /opt/omni-operator/llvm/lib/libLLVM-15.so /opt/omni-operator/lib/

The /opt/omni-operator, /opt/omni-operator/llvm, and /opt/omni-operator/lib directories can be customized.

Installing jemalloc

  1. Download https://github.com/jemalloc/jemalloc/archive/refs/tags/5.3.0.tar.gz and upload it to the management and compute nodes.
    1
2
3
    cd /opt/omni-operator/
tar zxvf jemalloc-5.3.0.tar.gz
mv jemalloc-5.3.0 jemalloc

    The /opt/omni-operator/jemalloc directory can be customized.

  2. Go to the jemalloc directory, run the script, and install the generated file.
    1
2
3
    cd jemalloc
./autogen.sh --disable-initial-exec-tls
make –j2
  3. Copy /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 to the /opt/omni-operator/lib directory.
    1
    cp /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 /opt/omni-operator/lib/
Parent topic: Installing the Feature