Architecture

OmniAdvisor uses AI algorithms to train optimal parameters of the big data engines Spark and Hive, improving Spark and Hive running efficiencies.

Both Spark and Hive have more than 100 parameters. These parameters have large value ranges, some of them are coupled with each other, and different parameter combinations have significant impact on performance. As a result, it is difficult to tune Spark and Hive parameters. Manual parameter tuning has a low efficiency in a high-dimensional space, and it is hardly possible to obtain an optimal set of parameters in a manual manner.

OmniAdvisor is an online parameter tuning tool for Spark SQL and application tasks and Hive SQL tasks. It parses historical task logs and uses AI algorithms to sample and tune parameters. In this way, it automatically recommends parameters for sampled tasks, providing end-to-end parameter tuning for online systems. OmniAdvisor consists of the following modules:

• Log parsing module: parses Spark and Hive logs to obtain task parameter information.
• Parameter sampling tuning module: samples task parameters of different configurations and uses the sampled parameters to execute tasks.
• Parameter recommendation module: For a task that has been tuned, you can search the database for the optimal parameters of historical tasks and use the optimal parameters as the execution parameters of the tuned task.
• Exception parameter processing module: If an exception occurs when a sampled or recommended SQL parameter is being executed, the exception must be handled.

Figure 2 illustrates a typical scenario of parameter tuning with OmniAdvisor.

Tuning procedure:

1. Use the default parameters to execute a Spark or Hive Tez task.
2. After the task is executed, the cluster retains the task execution log information (you need to configure spark.history.fs.logDirectory for the Spark cluster, or start the timeline server service for the Hive Tez engine). Use the log parsing module to parse the log information after the task is executed. Save the analyzed task parameter information, task execution status, and execution time to the MySQL database.
3. Select the list of tasks whose parameters you want to tune, obtain the historical parameters of each task from the database, sample the historical parameters using AI algorithms, obtain the target parameters, parse the execution results, and update optimal configuration information in the database.
4. When you need to re-execute a task, you can search the database for the optimal parameters of historical tasks and use them as the execution parameters of the task.