(Optional) Memory Borrowing

If off-heap memory has been configured in the startup parameters, skip this section. If not, perform the following steps:

1. Copy the parse_para.sh script to the ${SPARK_HOME}/bin/ directory.
2. Modify the permission on parse_para.sh.

   chmod 550 ${SPARK_HOME}/bin/parse_para.sh

3. Open the spark-submit file.

   cd ${SPARK_HOME}/bin/
   vi spark-submit

4. Press i to enter the insert mode and modify the last line.

   exec 3< <(bash "${SPARK_HOME}/bin/parse_para.sh" "$@")
   mapfile -t new_args <&3
   pid=$!
   wait $pid
   ret=$?
   if [  $ret -eq 2 ]; then
           exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
   else
           exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "${new_args[@]}"
   fi

5. Press Esc, type :wq!, and press Enter to save the file and exit.
6. Add startup parameters.

   --conf spark.omni.enableBorrow=true --conf spark.omni.borrowMemory.Fraction=90

   Table 1 Parameter description

   Parameter Name	Default Value	Description
   spark.omni.enableBorrow	false	Indicates whether to enable memory borrowing. true: yes; false: no.
   spark.omni.borrowMemory.Fraction	90	Percentage of the memory borrowed from the heap. The default value is 90%. If an error message is displayed indicating that the heap memory is insufficient, decrease the value of this parameter. If an error message is displayed indicating that the off-heap memory is insufficient, increase the value of this parameter.

   

   • If spark.memory.offHeap.enabled has already been set to true, memory borrowing cannot be enabled. To enable memory borrowing in this case, set this parameter to false.

     --conf spark.memory.offHeap.enabled=false

   • The executor-memory setting in the spark-default.conf file cannot be read.
   • If the executor-memory size has not been configured, memory borrowing cannot be enabled.