Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Starting OmniShuffle

  • You can start the OmniShuffle process directly using a script or start the process through Yarn. The direct startup mode is recommended.
  • When you use Yarn to start the process, configure LinuxContainerExecutor for Yarn. For details, see Hadoop official configuration. The official configuration requires that SUID be enabled, which may bring security risks. Evaluate possible security risks by yourself before enabling SUID.

Directly Starting OmniShuffle Using SmartKit

If the interval between stopping and starting OCK is longer than two minutes, you do not need to manually clear /meta and /mf.

  1. Access ZooKeeper.
    1
    		 
    zkCli.sh
  2. Check whether ZooKeeper has automatically deleted /meta and /mf.
    1
2
    		 
    ls /mf
ls /meta

    If the directories are not automatically deleted, you need to manually delete them.

    1
2
    		 
    deletall /mf
deletall /meta
  3. Configure the service flow. Select Run Command and import the start.json file. For details about the JSON file, see start.json. Copy the content in "JSON File Description" and save it as the start.json file.

  4. Click Run Service.

  5. Check that the execution is successful.

Starting OmniShuffle Using Yarn

Start OmniShuffle independently on the management node. Yarn manages the OmniShuffle lifecycle of the compute nodes.

  1. Start OmniShuffle on the management node.

    Use the single-node script ock-start-ockd.sh provided by OmniShuffle to start OmniShuffle. The script is stored in $OCK_HOME/ucache/23.0.0/linux-aarch64/sbin/.

  2. Check the Yarn configuration.
    1. Check the ~/.hadooprc file of the ockadmin user. The Yarn startup user is ockadmin. 
      vim ~/.hadooprc
    2. Press i to enter the insert mode and check for the following content. If the content does not exist, add it to the file. 
      export HADOOP_USER_NAME=ockadmin
export HDFS_DATANODE_USER=ockadmin
export HDFS_NAMENODE_USER=ockadmin
export HDFS_SECONDARYNAMENODE_USER=ockadmin
export YARN_RESOURCEMANAGER_USER=ockadmin
export YARN_NODEMANAGER_USER=ockadmin
    3. For details about how to configure Yarn LinuxContainerExecutor, visit https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/SecureContainer.html.
    4. Press Esc, type :wq!, and press Enter to save the file and exit.
  3. Configure the OmniShuffle node information file on the compute nodes. If the file does not exist, create one.
    $OCK_HOME/conf/ock_node_list

    Configure the name of the node where OmniShuffle is started in the information file (excluding the management node).

    Example:

  4. Optional: To configure the retry times and interval when Yarn fails to start OmniShuffle, modify the ock-start-ockd-by-yarn.sh file. For details about the configuration items, see ock-start-ockd-by-yarn.sh.
  5. Modify the ock-launch-cluster.sh script.

    The path of the script is $OCK_HOME/ucache/23.0.0/linux-aarch64/sbin/ock-launch-cluster.sh. The following is an example startup script. For details about the configuration items, see ock-launch-cluster.sh.

    • Set ock_memory to the larger one between 110% of the MF memory and the sum of the MF memory plus 10 GB. OCK running memory is included in the value. If the memory specified by the ock_memory parameter is less than the MF memory, the OCK fails to be started.
    • When multiple Yarn applications are running on the same node, change the names of the .appid files to help identify which application is shut down using the stop script.
  6. Run the script as the OCK user to start OmniShuffle.
    sh $OCK_HOME/ucache/23.0.0/linux-aarch64/sbin/ock-launch-cluster.sh

    You can view the ock-launch-cluster.log file in ock/logs/ to check whether the OmniShuffle startup request is submitted successfully.