Using the Feature

To use the OmniShuffle feature to execute Spark services, you need to start the Spark SQL CLI.

Create an ock_spark.conf file in the /home directory. For details about the parameters in the file, see spark.conf.

Create a file.
```
vi /home/ock_spark.conf
```

Press i to enter the insert mode and add the following content to the file:

spark.master yarn
spark.task.cpus 1
spark.shuffle.compress true
spark.shuffle.spill.compress true
spark.rdd.compress true
spark.executor.extraClassPath     /home/ockadmin/opt/ock/jars/*
spark.driver.extraClassPath       /home/ockadmin/opt/ock/jars/*
spark.driver.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
spark.executor.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
spark.driver.extraLibraryPath   /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
spark.executor.extraLibraryPath /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
spark.shuffle.manager              org.apache.spark.shuffle.ock.OCKShuffleManager
spark.shuffle.ock.manager true
spark.blacklist.enabled true
spark.files.fetchFailure.unRegisterOutputOnHost true
spark.shuffle.service.enabled  false
spark.blacklist.application.fetchFailure.enabled true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.driver.maxResultSize 2g
spark.serializer                        org.apache.spark.serializer.KryoSerializer
spark.shuffle.ock.home /home/ockadmin/opt/ock
spark.shuffle.ock.version 24.0.0
spark.shuffle.ock.binaryType linux-aarch64
spark.sql.broadcastTimeout 3000
spark.sql.extensions           org.apache.spark.sql.execution.adaptive.ock.BoostTuningExtension
spark.sql.ock.autoConfig.enabled true
spark.sql.ock.autoConfig.history true
spark.sql.ock.autoConfig.globalRuntimePartition false
spark.sql.ock.autoConfig.sample false

Press Esc, type :wq!, and press Enter to save the file and exit.

Start the Spark SQL CLI.

The following is an example of the native Spark SQL startup command. You can adjust the values of the configuration items based on your requirements.

/usr/local/spark/bin/spark-sql --deploy-mode client --driver-cores 8 --driver-memory 40g --num-executors 30 --executor-cores 6 --executor-memory 35g --master yarn --conf spark.task.cpus=1 --conf spark.default.parallelism=600 --conf spark.sql.broadcastTimeout=500 --conf spark.sql.shuffle.partitions=600 --conf spark.sql.adaptive.enabled=true --database tpcds_bin_partitioned_orc_3

Start the SparkExtension plugin.

spark-sql --deploy-mode client --driver-cores 8 \
                               --driver-memory 40G \
                               --num-executors 24 \
                               --executor-cores 12 \
                               --executor-memory 25g \
                               --master yarn \
                               --conf spark.sql.codegen.wholeStage=false \
                               --jars /home/ockadmin/opt/ock/jars/* \
                               --properties-file /home/ock_spark.conf \
                               --database tpcds_bin_partitioned_orc_3

Check whether the OmniShuffle feature has taken effect.
If the command output contains "Connected to meta rpc server<**.**.**.**> successfully", OmniShuffle has taken effect.

Parent topic: OmniShuffle