使用omni-op 1.4.0版本,运行tpc-ds q1用例,在创建native operator的时候报错
t('forum.solved') 已解决
发表于2025-09-28 09:29:10

各位专家好,

问题如标题所示,烦请帮忙看一下是什么原因,非常感谢。

软件栈版本如下:

os: openEuler 24.03 (LTS-SP2)

spark3.1.1

hadoop3.3.5

java: bisheng-jdk1.8.0_342

OmniRuntime 24.0.RC1

omniop 1.4.0

具体操作步骤如下

1. 启动spark-sehll

/home/software/spark/bin/spark-shell --deploy-mode client --driver-cores 8 --driver-memory 40g --master yarn --executor-cores 12 --executor-memory 5g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=35g --num-executors 24 --conf spark.executor.extraJavaOptions=" -XX:+UseG1GC" --conf spark.locality.wait=0 --conf spark.network.timeout=600 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.skewedJoin.enabled=true --conf spark.sql.autoBroadcastJoinThreshold=100M --conf spark.sql.broadcastTimeout=600 --conf spark.sql.shuffle.partitions=600 --conf spark.sql.orc.impl=native --conf spark.task.cpus=1 --conf spark.driverEnv.LD_LIBRARY_PATH=/opt/omni-operator/lib --conf spark.driverEnv.LD_PRELOAD=/opt/omni-operator/lib/libjemalloc.so.2 --conf spark.driverEnv.OMNI_HOME=/opt/omni-operator --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-1.4.0.jar --conf spark.driver.extraLibraryPath=/opt/omni-operator/lib --driver-java-options -Djava.library.path=/opt/omni-operator/lib --conf spark.executorEnv.LD_LIBRARY_PATH=/opt/omni-operator/lib --conf spark.executorEnv.LD_PRELOAD=/opt/omni-operator/lib/libjemalloc.so.2 --conf spark.executorEnv.MALLOC_CONF=narenas:2 --conf spark.executorEnv.OMNI_HOME=/opt/omni-operator --conf spark.executor.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-1.4.0.jar --conf spark.executor.extraLibraryPath=/opt/omni-operator/lib --jars /opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar --jars /opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar --conf spark.omni.sql.columnar.fusion=false --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.OmniColumnarShuffleManager --conf spark.sql.codegen.wholeStage=false --conf spark.sql.extensions=com.huawei.boostkit.spark.ColumnarPlugin --conf spark.omni.sql.columnar.RewriteSelfJoinInInPredicate=true --conf spark.sql.execution.filterMerge.enabled=true --conf spark.omni.sql.columnar.dedupLeftSemiJoin=true --conf spark.omni.sql.columnar.radixSort.enabled=true

2. 创建q1所需的view

val store_returns = spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/store_returns")

val date_dim= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/date_dim")

val store= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/store")

val customer= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/customer")

store_returns .createOrReplaceTempView("store_returns ")

date_dim.createOrReplaceTempView("date_dim")

store.createOrReplaceTempView("store")

customer.createOrReplaceTempView("customer")

3. 运行q1

spark.sql("""

| with customer_total_return as

| (select sr_customer_sk as ctr_customer_sk

| ,sr_store_sk as ctr_store_sk

| ,sum(SR_FEE) as ctr_total_return

| from store_returns

| ,date_dim

| where sr_returned_date_sk = d_date_sk

| and d_year =2000

| group by sr_customer_sk

| ,sr_store_sk)

| select  c_customer_id

| from customer_total_return ctr1

| ,store

| ,customer

| where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2

| from customer_total_return ctr2

| where ctr1.ctr_store_sk = ctr2.ctr_store_sk)

| and s_store_sk = ctr1.ctr_store_sk

| and s_state = 'TN'

| and ctr1.ctr_customer_sk = c_customer_sk

| order by c_customer_id

| LIMIT 100 ;

| """).show()

4. 执行过程中报错,错误log如下。可以看到是创建native operator的时候,出错了。

2025-09-28 08:28:11,561 WARN scheduler.TaskSetManager: Lost task 91.0 in stage 12.0 (TID 2468) (oe-2403 executor 2): nova.hetu.omniruntime.utils.OmniRuntimeException: Error Code: OPERATOR_RUNTIME_ERROR

Reason: Unable to create aggregator 0 / 1

Stack: /home/data/yarn/local/usercache/appcache/application_1759015658355_0003/container_1759015658355_0003_01_000003/tmp/boostkit-omniop-java-binding-1.4.0-aarch64425222691135202693.so(_ZNK11omniruntime9exception13OmniException4whatEv+0x14) [0xffff26a470f4]

        /home/data/yarn/local/usercache/appcache/application_1759015658355_0003/container_1759015658355_0003_01_000003/tmp/boostkit-omniop-java-binding-1.4.0-aarch64425222691135202693.so(Java_nova_hetu_omniruntime_operator_OmniOperatorFactory_createOperatorNative+0xcc) [0xffff26a50814]

        [0xffff9c415a68]

        at nova.hetu.omniruntime.operator.OmniOperatorFactory.createOperatorNative(Native Method)

        at nova.hetu.omniruntime.operator.OmniOperatorFactory.createOperator(OmniOperatorFactory.java:73)

        at com.huawei.boostkit.spark.util.OmniAdaptorUtil$.getAggOperator(OmniAdaptorUtil.scala:317)

        at org.apache.spark.sql.execution.ColumnarHashAggregateExec.$anonfun$doExecuteColumnar$6(ColumnarHashAggregateExec.scala:300)

        at org.apache.spark.sql.execution.ColumnarHashAggregateExec.$anonfun$doExecuteColumnar$6$adapted(ColumnarHashAggregateExec.scala:280)

        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)

        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:915)

        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)

新人帖
发表于2025/09/28