各位专家好,
问题如标题所示,烦请帮忙看一下是什么原因,非常感谢。
软件栈版本如下:
os: openEuler 24.03 (LTS-SP2)
spark3.1.1
hadoop3.3.5
java: bisheng-jdk1.8.0_342
OmniRuntime 24.0.RC1
omniop 1.4.0
具体操作步骤如下
1. 启动spark-sehll
/home/software/spark/bin/spark-shell --deploy-mode client --driver-cores 8 --driver-memory 40g --master yarn --executor-cores 12 --executor-memory 5g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=35g --num-executors 24 --conf spark.executor.extraJavaOptions=" -XX:+UseG1GC" --conf spark.locality.wait=0 --conf spark.network.timeout=600 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.skewedJoin.enabled=true --conf spark.sql.autoBroadcastJoinThreshold=100M --conf spark.sql.broadcastTimeout=600 --conf spark.sql.shuffle.partitions=600 --conf spark.sql.orc.impl=native --conf spark.task.cpus=1 --conf spark.driverEnv.LD_LIBRARY_PATH=/opt/omni-operator/lib --conf spark.driverEnv.LD_PRELOAD=/opt/omni-operator/lib/libjemalloc.so.2 --conf spark.driverEnv.OMNI_HOME=/opt/omni-operator --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-1.4.0.jar --conf spark.driver.extraLibraryPath=/opt/omni-operator/lib --driver-java-options -Djava.library.path=/opt/omni-operator/lib --conf spark.executorEnv.LD_LIBRARY_PATH=/opt/omni-operator/lib --conf spark.executorEnv.LD_PRELOAD=/opt/omni-operator/lib/libjemalloc.so.2 --conf spark.executorEnv.MALLOC_CONF=narenas:2 --conf spark.executorEnv.OMNI_HOME=/opt/omni-operator --conf spark.executor.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-1.4.0.jar --conf spark.executor.extraLibraryPath=/opt/omni-operator/lib --jars /opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.4.0-aarch64.jar --jars /opt/omni-operator/lib/boostkit-omniop-bindings-1.4.0-aarch64.jar --conf spark.omni.sql.columnar.fusion=false --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.OmniColumnarShuffleManager --conf spark.sql.codegen.wholeStage=false --conf spark.sql.extensions=com.huawei.boostkit.spark.ColumnarPlugin --conf spark.omni.sql.columnar.RewriteSelfJoinInInPredicate=true --conf spark.sql.execution.filterMerge.enabled=true --conf spark.omni.sql.columnar.dedupLeftSemiJoin=true --conf spark.omni.sql.columnar.radixSort.enabled=true
2. 创建q1所需的view
val store_returns = spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/store_returns")
val date_dim= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/date_dim")
val store= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/store")
val customer= spark.read.format("parquet").load("hdfs://localhost:8020/tpcds_sf100_parquet_snappy/customer")
store_returns .createOrReplaceTempView("store_returns ")
date_dim.createOrReplaceTempView("date_dim")
store.createOrReplaceTempView("store")
customer.createOrReplaceTempView("customer")
3. 运行q1
spark.sql("""
| with customer_total_return as
| (select sr_customer_sk as ctr_customer_sk
| ,sr_store_sk as ctr_store_sk
| ,sum(SR_FEE) as ctr_total_return
| from store_returns
| ,date_dim
| where sr_returned_date_sk = d_date_sk
| and d_year =2000
| group by sr_customer_sk
| ,sr_store_sk)
| select c_customer_id
| from customer_total_return ctr1
| ,store
| ,customer
| where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
| from customer_total_return ctr2
| where ctr1.ctr_store_sk = ctr2.ctr_store_sk)
| and s_store_sk = ctr1.ctr_store_sk
| and s_state = 'TN'
| and ctr1.ctr_customer_sk = c_customer_sk
| order by c_customer_id
| LIMIT 100 ;
| """).show()
4. 执行过程中报错,错误log如下。可以看到是创建native operator的时候,出错了。
2025-09-28 08:28:11,561 WARN scheduler.TaskSetManager: Lost task 91.0 in stage 12.0 (TID 2468) (oe-2403 executor 2): nova.hetu.omniruntime.utils.OmniRuntimeException: Error Code: OPERATOR_RUNTIME_ERROR
Reason: Unable to create aggregator 0 / 1
Stack: /home/data/yarn/local/usercache/appcache/application_1759015658355_0003/container_1759015658355_0003_01_000003/tmp/boostkit-omniop-java-binding-1.4.0-aarch64425222691135202693.so(_ZNK11omniruntime9exception13OmniException4whatEv+0x14) [0xffff26a470f4]
/home/data/yarn/local/usercache/appcache/application_1759015658355_0003/container_1759015658355_0003_01_000003/tmp/boostkit-omniop-java-binding-1.4.0-aarch64425222691135202693.so(Java_nova_hetu_omniruntime_operator_OmniOperatorFactory_createOperatorNative+0xcc) [0xffff26a50814]
[0xffff9c415a68]
at nova.hetu.omniruntime.operator.OmniOperatorFactory.createOperatorNative(Native Method)
at nova.hetu.omniruntime.operator.OmniOperatorFactory.createOperator(OmniOperatorFactory.java:73)
at com.huawei.boostkit.spark.util.OmniAdaptorUtil$.getAggOperator(OmniAdaptorUtil.scala:317)
at org.apache.spark.sql.execution.ColumnarHashAggregateExec.$anonfun$doExecuteColumnar$6(ColumnarHashAggregateExec.scala:300)
at org.apache.spark.sql.execution.ColumnarHashAggregateExec.$anonfun$doExecuteColumnar$6$adapted(ColumnarHashAggregateExec.scala:280)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:915)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
各位专家好,
问题如标题所示,烦请帮忙看一下是什么原因,非常感谢。
软件栈版本如下:
具体操作步骤如下
1. 启动spark-sehll
2. 创建q1所需的view
3. 运行q1
4. 执行过程中报错,错误log如下。可以看到是创建native operator的时候,出错了。