Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

SQL 3 (I/O- and CPU-intensive)

Purpose

SQL 3 is an I/O- and CPU-intensive scenario. You can optimize Spark executor parameters and adjust I/O parameters for the optimal performance.

Procedure

  • Based on the actual environment, adjust the number of running cores and memory size specified by Spark-Test-Tool in the script/spark-default.conf file to achieve the optimal performance. For example, for the Kunpeng 920 5220 processor, the following executor parameters are recommended for SQL 3.
    1
2
3
4
    yarn.executor.num 30
yarn.executor.cores 6
spark.executor.memory 24G
spark.driver.memory 36G
  • Adjust the I/O prefetch value. sd$i indicates the names of all Spark drives.
    1
    echo 4096 > /sys/block/sd$i/queue/read_ahead_kb
  • Use the default values defined in Hardware Tuning for other parameters.