EN
注册
我要评分
文档获取效率
文档正确性
内容完整性
文档易理解
在线提单
论坛求助
鲲鹏小智

使用OmniShuffle Shuffle加速特性

使用OmniShuffle特性执行Spark引擎业务,需通过启动Spark-SQL命令行窗口来执行。

  1. “/home”目录下新建“ock_spark.conf”文件,关于ock_spark.conf文件中的参数说明请参考spark.conf
    1. 新建文件。
      vi /home/ock_spark.conf
    2. “i”进入编辑模式,在文件中添加如下内容。
      spark.master yarn
      spark.task.cpus 1
      spark.shuffle.compress true
      spark.shuffle.spill.compress true
      spark.rdd.compress true
      spark.executor.extraClassPath     /home/ockadmin/opt/ock/jars/*
      spark.driver.extraClassPath       /home/ockadmin/opt/ock/jars/*
      spark.driver.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
      spark.executor.extraJavaOptions -Djava.library.path=/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars
      spark.driver.extraLibraryPath   /home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
      spark.executor.extraLibraryPath /home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/ucx/ucx:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/datakit:/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/mf:/home/ockadmin/opt/ock/jars:.
      spark.shuffle.manager              org.apache.spark.shuffle.ock.OCKShuffleManager
      spark.shuffle.ock.manager true
      spark.blacklist.enabled true
      spark.files.fetchFailure.unRegisterOutputOnHost true
      spark.shuffle.service.enabled  false
      spark.blacklist.application.fetchFailure.enabled true
      spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
      spark.driver.maxResultSize 2g
      spark.serializer                        org.apache.spark.serializer.KryoSerializer
      spark.shuffle.ock.home /home/ockadmin/opt/ock
      spark.shuffle.ock.version 23.0.0
      spark.shuffle.ock.binaryType linux-aarch64
      spark.sql.broadcastTimeout 3000
      spark.sql.extensions           org.apache.spark.sql.execution.adaptive.ock.BoostTuningExtension
      spark.sql.ock.autoConfig.enabled true
      spark.sql.ock.autoConfig.history true
      spark.sql.ock.autoConfig.globalRuntimePartition false
      spark.sql.ock.autoConfig.sample false
    3. “Esc”键,输入:wq!,按“Enter”保存并退出编辑。
  2. 启动Spark-SQL命令行窗口。

    原生Spark-SQL启动命令示例如下,可根据实际情况对配置项取值进行调整之类。

    /usr/local/spark/bin/spark-sql --deploy-mode client --driver-cores 8 --driver-memory 40g --num-executors 30 --executor-cores 6 --executor-memory 35g --master yarn --conf spark.task.cpus=1 --conf spark.default.parallelism=600 --conf spark.sql.broadcastTimeout=500 --conf spark.sql.shuffle.partitions=600 --conf spark.sql.adaptive.enabled=true --database tpcds_bin_partitioned_orc_3
    SparkExtension插件启动命令如下。
    spark-sql --deploy-mode client --driver-cores 8 \
                                   --driver-memory 40G \
                                   --num-executors 24 \
                                   --executor-cores 12 \
                                   --executor-memory 25g \
                                   --master yarn \
                                   --conf spark.sql.codegen.wholeStage=false \
                                   --jars /home/ockadmin/opt/ock/jars/* \
                                   --properties-file /home/ock_spark.conf \
                                   --database tpcds_bin_partitioned_orc_3
  3. 查看OmniShuffle是否生效。

    启动命令执行后出现Connected to meta rpc server<**.**.**.**> successfully则生效。

搜索结果
找到“0”个结果

当前产品无相关内容

未找到相关内容,请尝试其他搜索词