---
title: 执行Spark引擎业务
description: "使用OmniShuffle Shuffle加速组件特性执行Spark引擎业务，需通过启动Spark-SQL命令行窗口来执行。"
url: https://www.hikunpeng.com/document/detail/zh/kunpengboostkithistory/240RC5/bds/kunpengbds_omniruntime_20_0307.html
sourcePath: /source/zh/kunpengboostkithistory/240RC5/bds/kunpengbds_omniruntime_20_0307.html
indexId: 4b35643134ca7fd843c49fda6a680a9d01f53edec087a8aa19089bdb4be75d2080
---
# 执行Spark引擎业务

使用OmniShuffle Shuffle加速组件特性执行Spark引擎业务，需通过启动Spark-SQL命令行窗口来执行。

1. 在“/home”目录下新建“ock_spark.conf”文件，关于ock_spark.conf文件中的参数说明请参考spark.conf。
  a. 新建文件。

```
vi /home/ock_spark.conf
```


  b. 按“i”进入编辑模式，在文件中添加如下内容。

```
spark.task.cpus 1
spark.shuffle.compress true
spark.shuffle.spill.compress true
spark.rdd.compress true
spark.executor.extraClassPath     /home/ockadmin/opt/ock/jars/*
spark.driver.extraClassPath       /home/ockadmin/opt/ock/jars/*
spark.driver.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl
spark.executor.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl
spark.driver.extraLibraryPath   /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:.
spark.executor.extraLibraryPath /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt /ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:.
spark.driver.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
spark.executor.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
spark.driver.extraLibraryPath   /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
spark.executor.extraLibraryPath /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
spark.shuffle.manager              org.apache.spark.shuffle.ock.OCKRemoteShuffleManager
spark.shuffle.ock.manager true
spark.shuffle.ock.home /home/ockadmin/opt/ock
spark.shuffle.ock.version 24.0.0
spark.shuffle.ock.binaryType linux-aarch64
spark.executorEnv.HCOM_CONNECTION_RECV_TIMEOUT_SEC 30
spark.blacklist.enabled true
spark.files.fetchFailure.unRegisterOutputOnHost true
spark.shuffle.service.enabled  false
spark.blacklist.application.fetchFailure.enabled true
spark.serializer                        org.apache.spark.serializer.KryoSerializer
```


  c. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。
- ock_spark.conf配置文件可放在任意目录下，只需启动Spark-SQL指定该配置文件即可。
- “/home/ockadmin/opt/ock”需根据实际安装目录替换。
- spark.shuffle.manager 是必须项，需根据模式进行修改，RSS为 org.apache.spark.shuffle.ock.OCKRemoteShuffleManager。


2. 启动Spark-SQL命令行窗口。

  - 原生Spark-SQL启动命令示例如下，可根据实际情况对配置项取值进行调整之类。

```
/usr/local/spark/bin/spark-sql --deploy-mode client --driver-cores 8 --driver-memory 40g --num-executors 30 --executor-cores 6 --executor-memory 35g --master yarn --conf spark.task.cpus=1 --conf spark.default.parallelism=600 --conf spark.sql.broadcastTimeout=500 --conf spark.sql.shuffle.partitions=600 --conf spark.sql.adaptive.enabled=true --database tpcds_bin_partitioned_orc_3
```


  - SparkExtension插件启动命令如下。

```
spark-sql --deploy-mode client --driver-cores 8 \
--driver-memory 40G \
--num-executors 24 \
--executor-cores 12 \
--executor-memory 25g \
--master yarn \
--conf spark.sql.codegen.wholeStage=false \
--jars /home/ockadmin/opt/ock/jars/* \
--properties-file /home/ock_spark.conf \
--database tpcds_bin_partitioned_orc_3
```


3. 查看OmniShuffle Shuffle加速组件是否生效。
  启动命令执行后出现Shuffle initialize success.则生效。

  图1 返回信息

4. 执行SQL任务可以正常完成。
  图2 执行SQL任务

  图3 返回信息