Rate This Document
Findability
Accuracy
Completeness
Readability

Starting the ESS Mode

After the ESS mode is started, you can view the startup result on the ResourceManager WebUI and submit a Spark SQL task to verify it.

Prerequisites

You have completed the operations described in Preparations.

Procedure

  1. Configure a partition for the CapacityScheduler.

    Remove the RSS configuration items from the capacity-scheduler.xml file and distribute the updated file to all nodes.

  2. Leave the ESS compute node label blank, that is, use the default partition. See Figure 1.
    yarn rmadmin -replaceLabelsOnNode "agent02="
    Figure 1 ESS compute node
  3. Complete the necessary configurations before starting the ESS.
    1. Modify the /home/ockadmin/opt/ock/conf/ock.conf file to change the OCK mode to the ESS mode.
      ock.ucache.rss.mode = false
    2. In the Yarn startup script /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/sbin/ock-launch-cluster.sh, set ock_memory to a value greater than ock.mf.mem_size in the mf.conf file, and leave the partition label blank.
      An example of ock-launch-cluster.sh:
      # If you want to set the Yarn partition label of the launch server to ESS, leave this parameter blank.
      ock_master_partition_label=""
      ...
      # Memory space occupied by OCK, in MB.
      ock_memory="61440"

      An example of mf.conf:

      # Memory space occupied by MF, in bits.
      ock.mf.mem_size = 53687091200
    3. Write the host names of all nodes to /home/ockadmin/opt/ock/conf/ock_node_list. For example:
      agent01
      agent02
      agent03

      After the modification is complete, distribute the new file to all nodes.

  4. Run the startup script.
    sh /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/sbin/ock-launch-cluster.sh

    As shown in Figure 2, the ESS mode is started.

    Figure 2 Startup result

    The log indicates that the OCK compute node is started successfully.

    Figure 3 Log information
  5. Submit a Spark SQL task.

    If the task is executed successfully, the following information is displayed:

    Figure 4 Command output