Tuning Spark Application Parameters

Purpose

Based on the basic Spark configurations, use theoretical formulas to properly set the executor parameters and improve Spark application performance.

Procedure

• If you use Spark-Test-Tool to test scenarios SQL 1 to SQL 10, open the script/spark-default.conf file in the tool directory and add the following configuration items:

  1 2 3 4

  yarn.executor.num 15 yarn.executor.cores 19 spark.executor.memory 44G spark.driver.memory 36G

• If you use HiBench to test the WordCount, TeraSort, Bayesian, or K-means scenario, open the conf/spark.conf file in the tool directory, adjust the number of running cores and memory size based on the actual environment, and add the following configuration items:

  1 2 3 4

  yarn.executor.num 15 yarn.executor.cores 19 spark.executor.memory 44G spark.driver.memory 36G