Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Architecture

OmniAdvisor 2.0 consists of three parts: interception component, background tuning service, and historical database.

The interception component acts as the client, the background tuning service acts as the server, and the historical database stores data generated by the software. Figure 1 shows the OmniAdvisor 2.0 software.

Figure 1 OmniAdvisor 2.0

The interception component works as follows:

  1. Intercepts the tenant's task request and parses it to obtain task request information.
  2. Queries the historical database to obtain the optimal task configuration.
  3. Submits the task load and optimal configuration to the Spark cluster for execution and obtains the execution result.
  4. Records the execution result into the historical database.
The background tuning service works as follows:
  1. Waits for the administrator to specify the task load and submit a tuning request.
  2. Queries the historical database to obtain related historical data.
  3. Tunes parameters to obtain the optimal configuration and write it to the historical database.
    • By calling the standard REST APIs provided by the Spark history server and using the listening port specified by spark.history.rest.url in the common_config.ini configuration file, this feature collects detailed running metrics (such as task duration, resource usage, and stage execution status) during Spark SQL job execution in real time. After that, it stores the collected data in a structured manner into a user-preconfigured PostgreSQL database. The collected Spark SQL service runtime data is used solely for job performance tuning and is not disclosed to any third parties. This ensures data isolation and security throughout its lifecycle.
    • Before enabling this feature, technical support personnel must proactively obtain a clear written authorization (for example, through an email or a work order), or a trace data collection authorization statement, from the user or the user's authorized representative to ensure compliance with the user's data usage specifications. Do not enable the data collection function or access related data without explicit user authorization.
Parent topic: Feature Overview