Spark Automatic Tuning Example

Prerequisites

• Spark 3.3.1 has been configured in the environment. The application folder is located at /opt/tools/installed/spark-3.3.1-bin-hadoop3.2/.
• The Kunpeng AutoTuner has been installed. This section uses an RPM package as an example.

Procedure

1. Generate a simple template.

   devkit kat template -g -o /opt/template

   

   The -g parameter displays the interactive user interface. Set the basic parameters required by a template file. The -o /opt/template parameter indicates the path to the generated template file.

2. Select Spark.

   Press the ↑ or ↓ key to select an application, and press Enter to confirm the selection.

   Figure 1 Selecting an application
   
3. Select an application version.

   Press Enter to go to the version list and select the required version. Press Space to switch between the options in [], where Y indicates that the version is selected.

   Figure 2 Selecting a version (1)
   
   Figure 3 Selecting a version (2)
   
4. Select a parameter type.

   Press Esc to return to the upper-level directory, press ↓ to select ParamSpaces, and press Enter to go to the parameter type list.

   Spark indicates the application parameter, System indicates the system parameter, and Kunpeng uarch indicates the microarchitecture parameter.

   Figure 4 Selecting application parameters
   

   Press Space to switch between the options in [ ], where the option Y indicates that the parameter is enabled. In the example, Spark and System are selected.

   Figure 5 Selecting parameter types
   
5. Select detailed parameters.

   After selecting a parameter, press Enter to go to the parameter setting screen and select detailed parameters. The application and system parameters can be customized. You can enter / to enter the search mode and view the detailed parameters. You can also press Page Up and Page Down to view them.

   Figure 6 Selecting application parameters
   
6. Save the configuration to generate a simple template file.

   After the configuration is complete, press s to save the simple template file to the /opt/template/ directory. A folder named template_Spark_20250518_023644 is generated in the /opt/template/ directory. The folder contains the task parameter file task_Spark_System.yaml and application parameter file param_Spark_System.yaml.

   

   Press any key to return to the screen before saving the configuration.

   Command output:

   1	Spark configuration file saved successfully: {'path': '/opt/template/template_Spark_20250518_023644'}

7. Exit the interactive user interface.

   Press q to choose to exit, and then press y to confirm exiting. The directory for saving the template file is displayed on the terminal.

   Command output:

   1	[2025-05-15 06:38:31 UTC] [KAT] [message] - The path of the saved file is as follows: ['/opt/template/template_Spark_20250518_023644']

8. View the generated simple template file.

   ls /opt/template/template_Spark_20250518_023644/

   Command output:

   1	param_Spark_System.yaml task_Spark_System.yaml

   

   • Task parameter file (task_Spark_System.yaml): contains the host, application path, and pressure test information required for executing a task.
   • Application parameter file (param_Spark_System.yaml): contains detailed information about the parameters in each selected parameter space.

   You can view or modify the selected application and system parameters in the param_Spark_System.yaml file in the template file directory.

9. View the application parameter file.

   cat /opt/template/template_Spark_20250518_023644/param_Spark_System.yaml

   The param_xxxx.yaml file contains the application parameters (including the default parameter) selected on the interaction user interface. The application parameter file has the following content:

   

   

   Retain the default values for all parameters in the application parameter template file.

10. View and complete the task parameter file.

    vim /opt/template/template_Spark_20250518_023644/task_Spark_System.yaml

    Set the mandatory task parameters in the template file based on your requirements. For details about the parameters in this example, see Table 1. You can view the parameter description in the YAML file of the task template:

    

    Table 1 task_Spark_System.yaml file

    Parameter	Description
    configs/rounds	Total number of tuning rounds, which defaults to 50 and is 20 in this example.
    app/test_tool/tpcds_sql_index_list	Mandatory. List of performance test cases. The elements in the list are numbered from 1 to 99, and the suffix a or b must be contained in 14, 23, 24, and 39. Enter 23a in this example.
    app/test_tool/database_name	Mandatory. Name of the database used in the performance test. Set it to tpcds_orc_hive_3 in this example.
    app/host_name	Name of the server on which the application is to be executed. If this parameter is not set, the task is executed on the local server by default.
    app/test_tool/app_exe_path	Mandatory. Path to the folder where the Spark executable file is located. Set it to /opt/tools/installed/spark-3.3.1-bin-hadoop3.2/bin in this example.

11. Save the task parameter file and exit.

    After completing the basic configuration, press Esc, input :wq!, and press Enter to save the file and exit.

12. View the function manual.

    1	devkit kat man --lang en

    Figure 7 Manual directory
    

    Type 1 and press Enter to view the Kunpeng AutoTuner introduction.

    Figure 8 quick start
    

    Enter q to exit the current page and return to the selection page. Enter q to exit the function manual.

13. Enable automatic tuning.

    Use the completed simple template file to start automatic tuning.

    devkit kat train -t /opt/template/template_Spark_20250518_023644/task_Spark_System.yaml -p /opt/template/template_Spark_20250518_023644/param_Spark_System.yaml

    Figure 9 Automatic tuning result
    
    

    • The Spark performance metric is latency, with a baseline of 126 seconds. After 20 rounds of automatic tuning, the latency is reduced to 53 seconds, representing a 58% reduction.
    • The directory for storing the tuning result file is provided at the end of the command output. By default, the file is generated in the current directory (/home/template/DevKit-CLI-x.x.x-Linux-Kunpeng).
14. Check the tuning result.

    The parameter set generated after automatic tuning is generated in the tool directory, which is /home/template/DevKit-CLI-x.x.x-Linux-Kunpeng/train-20250520-091026 in the example. You can select a parameter set based on the tuning result to adjust application parameters.

15. Use the tuning result.

    devkit kat use -i /home/template/DevKit-CLI-x.x.x-Linux-Kunpeng/train-20250520-091026

    Go to the interactive user interface.

16. View the use subcommand function manual.

    Type man --lang en and press Enter to view the use subcommand manual directory. Select the content to be viewed as required.

    Figure 10 use subcommand manual directory
    

    Type 1 and press Enter to view the help information about the show subcommand.

    Figure 11 Content of the show subcommand
    

    Enter q to exit the current page and return to the selection page. Enter q to exit the function manual.

17. View the performance test data.

    Type show and press Enter to view the automatic tuning statistics.

    Figure 12 Tuning result
    

    Type show top 5 and press Enter to view the five rounds with the best performance data.

    Figure 13 Top 5 performance data
    
18. Perform a test again.

    Before assigning values to parameters, you can retest the rounds with good performance improvement for multiple times to check whether the tuned parameters are stable. A smaller value of CV(%) (CV: coefficient of variation) indicates higher performance data consistency and more stable parameters in a round. That means you can consider using the parameters of that round to the actual application.

    In a proper test scenario, a CV(%) value below 1% indicates that the parameter tuning effect is very stable. A value between 1% and 5% indicates good stability. When the CV(%) value falls between 5% and 10%, the effect is less stable, and you may choose whether to accept it based on your requirements. If the CV(%) value exceeds 10%, the effect is considered unstable, and it is recommended to address potential sources of data variability, such as environmental or human factors.

    

    Generally, A/B testing is used to ensure stable test results. For example, to retest the result of the round (assume it is round A) whose performance ranks first, you can run test 0,A 5 to alternately test five groups of the baseline round and highest-performing for 10 times.

    Type test 11 5 and press Enter to retest the round (the 11th round of automatic tuning) that delivers the best performance data for five times.

    Figure 14 Retest data
    
    

    In the example, after the specified round has been tested for five times, the final CV(%) value is only 3.94%, indicating that the performance data obtained using the parameters of this round is consistent and the tuned parameters are stable. Therefore, these parameters can be used in the application and system.

19. View the importance of all parameters.

    Type show importance and press Enter.

    Figure 15 Importance of all parameters
    

    A higher value in the importance(%) column indicates stronger correlation between the parameter and the tuning effect.

20. Use the optimal parameters inferred from parameter importance analysis to perform tests.

    Type ibesttest and press Enter.

    Figure 16 Using the optimal parameters for tests
    
21. Resume automatic training.

    Type train -r 5 and press Enter to perform the next five rounds of automatic tuning on the use screen.

    Figure 17 Resuming automatic training
    
22. Apply the parameters of a specified round to the application and system.

    If you are satisfied with the performance data of the 11th round, type set 11 and press Enter to set the parameters of the 11th round of automatic tuning to the application and system.

    If the value assignment is successful, the following information is displayed:

    1	[2025-05-19 06:55:48 UTC] [KAT] [message] - set params of round 11 success!

23. Exit the interactive user interface.

    Type exit and press Enter.