OmniStream编译手册

1、操作系统要求

OS：OpenEuler22.03 LTS SP3

2、安装依赖组件

GCC
Cmake
JDK
maven
zlib
LLVM
googletest
jemalloc
nlomann json
libboundscheck
snappy
rocksdb
xxHash
librdkafka
BoostKit-kaccjson
BoostKit-ksl
OmniOperator

说明：以下的组件都安装在/opt/buildtools/目录下; $PARALLELISM表示编译指定并行度。

通过yum直接安装部分依赖（GCC, zlib）。

yum install -y sudo patch glibc zip unzip zlib wget curl tar gcc java gcc-c++ make rpm git python3 cpio cmake autoconf automake gettext-devel libtool meson openssl-devel libuuid-devel python3-pyelftools python-pyelftools gcc-c++ libaio-devel ncurses-devel CUnit-devel libiscsi-devel json-c-devel libcmocka-devel python3-pip python3-requests byacc flex binutils-devel bison boost boost-devel python3-devel doxygen dos2unix google-benchmark google-benchmark-devel libasan glibc-devel libasan-static jq bsdtar expect openssh vim strace python-jenkins python-concurrent-log-handler python3-gevent python3-marshmallow python3-pyyaml python-pandas python-xlrd python-retrying python-esdk-obs-python rpm-build

安装JAVA。

mkdir -p /opt/buildtools/openjdk8
cd /opt/buildtools/openjdk8 && wget https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u462-b08/OpenJDK8U-jdk_aarch64_linux_hotspot_8u462b08.tar.gz
tar -zxvf OpenJDK8U-jdk_aarch64_linux_hotspot_8u462b08.tar.gz
rm -rf OpenJDK8U-jdk_aarch64_linux_hotspot_8u462b08.tar.gz

安装Cmake。

cd /opt/buildtools/ && wget https://github.com/Kitware/CMake/releases/download/v3.28.2/cmake-3.28.2-linux-aarch64.tar.gz
tar -zxvf cmake-3.28.2-linux-aarch64.tar.gz
rm -rf cmake-3.28.2-linux-aarch64.tar.gz
rm -rf /usr/local/bin/cmake
ln -sf cmake-3.28.2-linux-aarch64/bin/cmake /usr/local/bin/cmake 
ln -sf cmake-3.28.2-linux-aarch64/share/cmake-3.28 /usr/local/share/

安装maven。

mkdir -p /opt/buildtools/apache-maven
cd /opt/buildtools/apache-maven && wget https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.8.3/apache-maven-3.8.3-bin.zip
unzip apache-maven-3.8.3-bin.zip
rm -rf apache-maven-3.8.3-bin.zip

安装LLVM。

mkdir -p /opt/buildtools/LLVM-15.0.4
export LLVM_install_dir=/opt/buildtools/LLVM-15.0.4
cd /opt/buildtools/LLVM-15.0.4 && wget https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-15.0.4.tar.gz
tar -zxvf llvm-project-llvmorg-15.0.4.tar.gz
rm -rf llvm-project-llvmorg-15.0.4.tar.gz
mkdir -p /opt/buildtools/LLVM-15.0.4/llvm-project-llvmorg-15.0.4/build
cd /opt/buildtools/LLVM-15.0.4/llvm-project-llvmorg-15.0.4/build
cmake -G "Unix Makefiles" -DLLVM-TARGETS_TO_BUILD="host;ARM;X86;AArch64;BPE" -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=true -DLLVM_ENABLE_RTTI=ON -DLLVM_ENABLE_PROJECTS="clang;lld;libunwind;compiler-rt;lldb" -DCMAKE_INSTALL_PREFIX=/usr/local/ ../llvm
make -j $PARALLELISM
make install

安装googletest。

cd /opt/buildtools
git clone https://github.com/google/googletest.git
cd googletest && git checkout tags/v1.14.0
cmake CMakeLists.txt
make -j$PARALLELISM
make install

安装jemalloc。

cd /opt/buildtools
git clone https://github.com/jemalloc/jemalloc.git
cd jemalloc && git checkout tags/5.3.0
./autogen.sh --disable-initial-exec-tls --with-lg-page=16
make -j$PARALLELISM && make install

安装nlomann Json。

cd /opt/buildtools
git clone https://github.com/nlohmann/json.git -b v3.11.3
cd json && mkdir build && cd build
cmake ..
make -j$PARALLELISM && make install

安装snappy。

cd /opt/buildtools
git clone https://gitee.com/src-openeuler/snappy.git 
cd snappy && git checkout tags/openEuler-24.03-LTS-SP1-release
tar -zxvf snappy-1.1.10.tar.gz
rm -f snappy-1.1.10.tar.gz
cd snappy-1.1.10 && patch -p1 < ../add-option-to-enable-rtti-set-default-to-current-ben.patch && patch -p1 < ../remove-dependency-on-google-benchmark-and-gmock.patch
mkdir -p build && cd build
cmake -DSNAPPY_BUILD_BENCHMARKS=OFF -DSNAPPY_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON  .. 
make -j$PARALLELISM && make install

安装rocksdb。

cd /opt/buildtools
git clone https://github.com/facebook/rocksdb.git
cd rocksdb && git checkout tags/v8.11.4
mkdir -p build && cd build
cmake .. -DWITH_SNAPPY=1 -DCMAKE_BUILD_TYPE=Release -DUSE_RTTI=1 -DWITH_GFLAGS=0
make -j$PARALLELISM && make install

安装xxHash。

cd /opt/buildtools && git clone https://github.com/Cyan4973/xxHash.git
cd xxHash && git checkout tags/v0.8.2
mkdir -p build && cd build
cmake ../cmake_unofficial && cmake --build .
cmake --build . --target install

安装BoostKit-ksl。

cd /opt/buildtools
mkdir ksl && cd ksl
下载 https://www.hikunpeng.com/boostkit/library/system?subtab=kpglibc&bread=false
unzip BoostKit-ksl_2.5.1.zip && rpm -ivh boostkit-ksl-*.aarch64.rpm

3、编译OmniStream

cd /opt/buildtools
git clone https://gitee.com/openeuler/OmniStream.git
mkdir -p OmniStream/3rdparty/

编译前需要安装以下几个组件：

安装libboundscheck。

cd /opt/buildtools/OmniStream/3rdparty
git clone https://gitee.com/openeuler/libboundscheck.git
cd libboundscheck && git checkout tags/v1.1.16
make CC=gcc

安装BoostKit-kaccjson。

cd /opt/buildtools/OmniStream/3rdparty
mkdir Kacc-Json && cd Kacc-Json
下载组件 https://www.hikunpeng.com/boostkit/library/detail?subtab=%E9%B2%B2%E9%B9%8FJson%E5%BA%93
unzip BoostKit-kaccjson_1.0.0.zip

安装librdkafka。

cd /opt/buildtools/OmniStream/3rdparty
git clone https://github.com/confluentinc/librdkafka.git
cd librdkafka && git checkout tags/v2.6.1
git apply "/opt/buildtools/OmniStream/cpp/connector/kafka/omni_kafka_opt.patch"
./configure --CFLAGS="-O3" --CXXFLAGS="-O3"
make -j$PARALLELISM
make install

配置环境变量。
在/etc/profile文件中追加以下内容。

export JAVA_HOME=/opt/buildtools/openjdk8/jdk8u462-b08
export JRE_HOME=/opt/buildtools/openjdk8/jdk8u462-b08/jre
export MAVEN_HOME=/opt/buildtools/apache-maven/apache-maven-3.8.3
export LLVM_HOME=/opt/buildtools/LLVM-15.0.4 
export CMAKE_ROOT=/opt/buildtools/cmake-3.28.2-linux-aarch64/share
export PATH=/opt/buildtools/cmake-3.28.2-linux-aarch64/bin:/opt/buildtools/LLVM-15.0.4/bin:/opt/buildtools/apache-maven/apache-maven-3.8.3/bin:/opt/buildtools/openjdk8/jdk8u462-b08/bin:/opt/buildtools/openjdk8/jdk8u462-b08/jre/bin:$PATH
export CLASSPATH=/opt/buildtools/openjdk8/jdk8u462-b08/lib:$/opt/buildtools/openjdk8/jdk8u462-b08/jre/lib:$CLASSPATH

export C_INCLUDE_PATH=/opt/buildtools/OmniStream/3rdparty/libboundscheck/include:/opt/buildtools/OmniStream/3rdparty/opt/lib/include:/opt/buildtools/LLVM-15.0.4/include:$JAVA_HOME/include/:$JAVA_HOME/include/linux:/usr/local/include:/opt/buildtools/OmniStream/3rdparty:/usr/local/ksl/include:$C_INCLUDE_PATH 
export CPLUS_INCLUDE_PATH=/opt/buildtools/OmniStream/3rdparty/libboundscheck/include:/opt/buildtools/OmniStream/3rdparty/opt/lib/include:/opt/buildtools/LLVM-15.0.4/include:$JAVA_HOME/include/:$JAVA_HOME/include/linux:/usr/local/include:/opt/buildtools/OmniStream/3rdparty/Kacc-Json/include:/opt/buildtools/OmniStream/3rdparty:/usr/local/ksl/include:$CPLUS_INCLUDE_PATH
export LD_LIBRARY_PATH=/opt/buildtools/openjdk8/jdk8u462-b08/jre/lib/aarch64/server/:/opt/buildtools/OmniStream/3rdparty/opt/lib/:/usr/local/ksl/lib:/opt/buildtools/OmniStream/3rdparty/libboundscheck/lib:/opt/buildtools/LLVM-15.0.4/lib:/usr/local/lib:$LD_LIBRARY_PATH 
export LIBRARY_PATH=/opt/buildtools/OmniStream/3rdparty/opt/lib/:/usr/local/ksl/lib:/opt/buildtools/LLVM-15.0.4/lib:/usr/local/lib:/opt/buildtools/OmniStream/3rdparty/libboundscheck/lib:/opt/buildtools/OmniStream/3rdparty/Kacc-Json:/usr/local/lib:$LIBRARY_PATH

追加完内容后，请执行 source /etc/profile

安装OmniOperator。

cd /opt/buildtools/OmniStream/3rdparty
git clone https://gitee.com/openeuler/OmniOperator.git -b operator_flink_1107
cp -r /opt/buildtools/OmniStream/3rdparty/OmniOperator
/opt/buildtools/OmniStream/3rdparty/OmniOperatorJIT
cd OmniOperator/build_scripts
export OMNI_HOME=/opt/buildtools/OmniStream/3rdparty/OmniOperatorJIT
./build.sh release:java --exclude-test

编译详情请参考https://gitee.com/openeuler/OmniOperator。

编译完后，拷贝文件，步骤如下。

mkdir -p /opt/buildtools/OmniStream/3rdparty/opt/lib
mkdir -p /opt/buildtools/OmniStream/3rdparty/opt/java-binding
cp -r /opt/buildtools/OmniStream/3rdparty/OmniOperatorJIT/lib/* /opt/buildtools/OmniStream/3rdparty/opt/lib 
cp /opt/buildtools/OmniStream/3rdparty/OmniOperatorJIT/java-binding/libboostkit-omniop-java-binding-1.9.0-aarch64.so /opt/buildtools/OmniStream/3rdparty/opt/java-binding/

编译OmniAdaptor。

cd /opt/buildtools/
git clone https://gitee.com/openeuler/OmniAdaptor.git
cd OmniAdaptor/omnistream/omniop-flink-extension/omni-flink-bundle/
mvn clean package -DskipTests

编译OmniStream。其中，HOME变量表示operator安装目录

export HOME=/opt/buildtools/OmniStream/3rdparty
cd /opt/buildtools/OmniStream/cpp
mkdir build
cd build && cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$PARALLELISM

OmniStream结果文件路径如下。

/opt/buildtools/OmniStream/cpp/build/jni/libtnel.so
/opt/buildtools/OmniAdaptor/omnistream/omniop-flink-extension/java/target/flink-tnel-0.1-SNAPSHOT.jar

注意点

1、部分依赖组件由于服务器网络问题无法直接从网站下载，已上传到gitee仓库中，请自行下载，仓库地址为

https://gitee.com/one-more-bowl-at-most/omnistream_dependency

2、如果是网页申请资源，请确认申请服务器资源，如果资源规格是8U16G，请通知对应的工作人员进行扩容。

3、本文中的$PARALLELISM为实际使用的并行数，根据机器实际核数而定。

4、部署、验证请参考https://www.hikunpeng.com/document/detail/zh/kunpengboostkithistory/251RC1/bds/kunpengbds_omniruntime_20_0925.html ，其中，开源任务验证请参考“使能特性”中“SQL使能特性”。不用去安装UDF翻译工具、AI4C工具，开源任务没有涉及DataStream相关章节内容。

5、本地验证：将附件中check.zip解压获得的bid.csv和test_q0.sql放到服务器中，设置文件权限

chmod 750 bid.csv test_q0.sql

修改test_q0.sql中 “'path' = 'bid.csv’,”这一行，path的值修改为bid.csv在服务器上的绝对路径。通过使能omnistream和不使能omnistream，两者结果一致，则验证成功。

启动Flink，并执行test_q0.sql

cd Flink安装目录/bin
./start-cluster.sh
./sql-client -f “服务器上test_q0.sql的绝对路径”

6、社区任务开发与验证：修改源码目录中cpp/table/runtime/operators/join下的join算子文件，涉及到该目录下所有文件的修改。

如何验证改动是否正确：
使用附件join_task.zip文件解压，将q20._csv.sql改为outer join等其他任务join type，分别用flink不使能omnistream和使能omnistream两种方式，比较输出结果内容是否一致。（可以自行修改用例，或者将结果insert到表中，观察生成表是否一致）。其中q20_csv.sql中path的值和“1”中一样，修改为auction_join.csv和bid_join.csv中的实际路径。

7、使能OmniStream：

修改flink/bin目录下的config.sh，修改该文件中的constructFlinkClassPath函数，注释最后一行的 echo "FLINK_CLASSPATH" "FLINK_DIST"，在注释行下补充两行。
```
PATCH="flink-tnel-0.1-SNAPSHOT.jar的绝对路径"
echo $PATCH:"$FLINK_CLASSPATH" "$FLINK_DIST"
```
修改flink/conf目录下的flink-conf.yaml文件。修改属性env.java.opts，修改为
```
env.java.opts: -Djava.library.path="libtnel.so所在目录的绝对路径"
```

8、不使能OmniStream：将config.sh和flink-conf.yaml恢复为原始格式。

9、部署OmniStream需要的文件列表为：

flink-tnel-0.1-SNAPSHOT.jar
libboostkit-omniop-codegen-1.9.0-aarch64.so
libboostkit-omniop-operator-1.9.0-aarch64.so
libboostkit-omniop-vector-1.9.0-aarch64.so
libboundscheck.so
libtnel.so
libLLVM-15.so
libjemalloc.so.2
librdkafka.so.1
librocksdb.so.8
libsnappy.so.1
libxxhash.so.0

OmniStream编译手册

1、操作系统要求

2、安装依赖组件

3、编译OmniStream

注意点

关于鲲鹏

新闻与活动

交流与资讯

支持与服务

开源社区