Upgrading JDK and Tuning JVM Parameters
Purpose
The latest JDK version introduces performance optimizations for Spark. To fully leverage these improvements, the outdated JDK on the server must be replaced with the new version, and Spark should be configured to run with it.
Procedure
Add the following configuration items to the spark-defaults.conf configuration file of Spark to specify the new JDK path and optimization parameters.
1 2 3 4 | spark.executorEnv.JAVA_HOME /usr/local/jdk8u222-b10 spark.yarn.appMasterEnv.JAVA_HOME /usr/local/jdk8u222-b10 spark.executor.extraJavaOptions -XX:+UseNUMA -XX:BoxTypeCachedMax=100000 -XX:ParScavengePerStrideChunk=8192 spark.yarn.am.extraJavaOptions -XX:+UseNUMA -XX:BoxTypeCachedMax=100000 -XX:ParScavengePerStrideChunk=8192 |
Parent topic: General Tuning