Installing Dependencies
When installing OmniOperator locally, install the LLVM and jemalloc dependency packages on the management and compute nodes to combine OmniOperator and Spark.
In the Spark on Yarn scenario, use the --archives parameter of Spark to simplify the deployment. You can install the dependencies either by downloading pre-compiled SO files or by compiling source code. The precompiled SO file can be downloaded and installed quickly, and is suitable for most scenarios. In contrast, compiling and installing from source code is slower, but may be required for certain compliance scenarios.
Installing Dependencies (By Downloading Pre-compiled SO Files)
Installing LLVM and jemalloc
- The Dependency_library.zip package contains the Dependency_library_openeuler.zip and Dependency_library_centos.zip subpackages, which are used for openEuler and CentOS, respectively. The following uses openEuler as an example. To install OmniOperator on CentOS, replace Dependency_library_openeuler.zip in the following commands with Dependency_library_centos.zip.
- In the following steps, you can replace the /opt/omni-operator and /opt/omni-operator/lib with custom directories.
- Create an /opt/omni-operator/ directory on the management and compute nodes as the root directory for installing OmniOperator. Then go to the directory.
1 2 3
mkdir /opt/omni-operator cd /opt/omni-operator rm -rf Dependency_library_*.zip
- Upload the Dependency_library packages (Dependency_library.zip, Dependency_library.z01, and Dependency_library.z02) obtained from Obtaining Software Packages to the /opt/omni-operator/ directory and decompress them.
1 2
zip -F Dependency_library.zip --out Dependency_library_complete.zip unzip Dependency_library_complete.zip
- Create an /opt/omni-operator/lib directory, and copy the libLLVM-15.so and libjemalloc.so.2 subpackages in Dependency_library_openeuler to the /opt/omni-operator/lib directory.
If LLVM and jemalloc have been installed in the environment, delete the existing libLLVM-15.so and libjemalloc.so.2 files before running the copy command.
1 2
rm -rf /opt/omni-operator/lib/libjemalloc.so.2 rm -rf /opt/omni-operator/lib/libLLVM-15.so
1 2 3 4
cd /opt/omni-operator mkdir lib cp /opt/omni-operator/Dependency_library/Dependency_library_openeuler/libjemalloc.so.2 /opt/omni-operator/lib cp /opt/omni-operator/Dependency_library/Dependency_library_openeuler/libLLVM-15.so /opt/omni-operator/lib
Install Dependencies (By Compiling Source Code)
Installing LLVM
- Download llvm-project-llvmorg-15.0.4.tar.gz, create an /opt/omni-operator directory on the management and compute nodes as the root directory for installing OmniOperator, go to the directory, and upload the package to /opt/omni-operator.
1 2 3 4 5 6
mkdir /opt/omni-operator cd /opt/omni-operator tar zxvf llvm-project-llvmorg-15.0.4.tar.gz mv llvm-project-llvmorg-15.0.4 llvm cd llvm mkdir build
- Go to the build directory, and compile and install LLVM.
1 2 3 4
cd ./build cmake -DCMAKE_INSTALL_PREFIX=/opt/omni-operator/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=true -DLLVM_ENABLE_PROJECTS="clang" -G "Unix Makefiles" ../llvm make -j4 make install
- Create a lib directory in /opt/omni-operator and copy /opt/omni-operator/llvm/lib/libLLVM-15.so to /opt/omni-operator/lib.
1 2
mkdir /opt/omni-operator/lib cp /opt/omni-operator/llvm/lib/libLLVM-15.so /opt/omni-operator/lib/
The /opt/omni-operator, /opt/omni-operator/llvm, and /opt/omni-operator/lib directories can be customized.
Installing jemalloc
- Download jemalloc-5.3.0.tar.gz and upload it to the management and compute nodes.
1 2 3
cd /opt/omni-operator/ tar zxvf jemalloc-5.3.0.tar.gz mv jemalloc-5.3.0 jemalloc
The /opt/omni-operator/jemalloc directory can be customized.
- Go to the jemalloc directory, run the script, and install the generated file.
1 2 3
cd jemalloc ./autogen.sh --disable-initial-exec-tls make -j2
- Copy /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 to the /opt/omni-operator/lib directory.
1cp /opt/omni-operator/jemalloc/lib/libjemalloc.so.2 /opt/omni-operator/lib/