Starting and Configuring the Spark History Server

Before using the OmniAdvisor 2.0 feature, activate Anaconda and start the Spark history server. Ensure that the management node meets the requirements described in this section.

1. Activate Anaconda.

   conda activate OmniAdvisor2.0

2. In the sbin directory of Spark, start the Spark history server.

   sh $SPARK_HOME/sbin/start-history-server.sh

3. Verify that the Spark history server has been started.

   Enter http://<server1>:18080 or https://<server1>:18480 in the address box of the browser to view the UI. By default, port 18080 is used in HTTP mode and port 18480 is used in HTTPS mode.

   

4. Optional: Change the Spark history server port.

   The default port of the Spark history server is 18080. If the port is occupied, you can modify the configuration to change the port to another port that is not occupied. The following describes how to change the port to 18181.

   1. Open the $SPARK_HOME/conf/spark-defaults.conf configuration file.

      vi $SPARK_HOME/conf/spark-defaults.conf

   2. Press i to enter the insert mode and add spark.history.ui.port or change its value to 18181.

      spark.history.ui.port 18181

   3. Press Esc, type :wq!, and press Enter to save the file and exit.
   4. Restart the Spark history server.

      In the sbin directory of Spark, stop and start the history server.

      # Stop the Spark history server.
      sh $SPARK_HOME/sbin/stop-history-server.sh
      # Start the Spark history server.
      sh $SPARK_HOME/sbin/start-history-server.sh