Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Prerequisites

Before using the OmniAdvisor feature, activate or start associated services. Ensure that the management node meets the requirements described in this section.

Tuning Spark

  1. Activate Anaconda.
    1
    		 
    conda activate OmniAdvisor
  2. In the sbin directory of Spark, start the Spark history server.
    1
    		 
    sh start-history-server.sh
  3. Verify that the history server has been started.

    In the address box of the browser, enter http://<server1>:18080 (the default port is 18080) to view the UI.

  4. Optional: Change the history server port.

    The default history server port is 18080. If this port is occupied, modify the configuration file to replace this port with another one. For example, change this port to 18180:

    1. Open the $SPARK_HOME/conf/spark-defaults.conf configuration file. 
      vi $SPARK_HOME/conf/spark-defaults.conf
    2. Press i to enter the insert mode and add spark.history.ui.port or change its value to 18180. 
      1
      		 
      spark.history.ui.port 18180
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
    4. Restart the history server.
      In the sbin directory of Spark, stop and start the history server: 
      1
2
3
4
      		 
      # Stop the history server.
sh stop-history-server.sh
# Start the history server.
sh start-history-server.sh

      The IP address and port of the Spark history server must be added to the configuration file of the log parsing module.

Tuning Hive

  1. Activate Anaconda.
    1
    		 
    conda activate OmniAdvisor
  2. Start the timeline server.
    1
    		 
    yarn --daemon start timelineserver
  3. Verify that the timeline server has been started.

    In the address box of the browser, enter http://<server1>:8188 (the default port is 8188) to view the UI.

  4. Optional: Change the Yarn timeline server port.

    The default timeline server port is 8188. If this port is occupied, modify the configuration file to replace this port with another one. For example, change this port to 8288:

    1. Open the $HADOOP_HOME/etc/hadoop/yarn-site.xml configuration file. 
      vi $HADOOP_HOME/etc/hadoop/yarn-site.xml
    2. Press i to enter the insert mode and set the yarn.timeline-service.webapp.address field to ${yarn.timeline-service.hostname}:8288. 
      1
2
3
4
      		 
      <property>
    <name>yarn.timeline-service.webapp.address</name>
    <value>${yarn.timeline-service.hostname}:8288</value>
</property>
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
    4. Restart the Yarn timeline server. 
      1
2
3
4
      		 
      # Stop the timeline server.
yarn --daemon stop timelineserver
# Start the timeline server.
yarn --daemon start timelineserver

      The IP address and port of the Yarn timeline server must be added to the configuration file of the log parsing module.

  5. Modify the $HIVE_HOME/conf/hive-site.xml configuration file to ensure that Tez is specified for hive.execution.engine.
    1. Open the configuration file. 
      vi $HIVE_HOME/conf/hive-site.xml
    2. Press i to enter the insert mode and ensure that Tez is specified for hive.execution.engine. 
       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
      		 
      <property>
    <name>hive.execution.engine</name>
    <value>tez</value>
</property>
<property>
    <name>hive.cli.tez.session.async</name>
    <value>false</value>
    <description>
      Whether to start Tez
      session in background when running CLI with Tez, allowing CLI to be available earlier.
    </description>
 </property>
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
Parent topic: Using the Feature