Rate This Document
Findability
Accuracy
Completeness
Readability

Installing OmniShuffle

Perform the following operations on all nodes.

  1. Go to the nodes where you want to install OmniShuffle, and download the OmniShuffle software package from Table 3 to the /home directory.
    tar -zxvf /home/BoostKit-omnishuffle_1.3.0.tar.gz
    cd /home/BoostKit-omnishuffle_1.3.0
    tar -zxvf omnishuffleUtils_openeuler-aarch64.tar.gz
  2. Modify parameters in the common_params.sh file.
    1. Go to the omnishuffleUtils/common directory.
      cd omnishuffleUtils/common
    2. Open the common_params.sh file.
      vim common_params.sh
    3. Press i to enter the insert mode and modify the following content of the file:
      ock_root_path=/opt
      spark_version="spark-3.1"
    4. Press Esc, type :wq!, and press Enter to save the file and exit.
  3. Go to the bin directory and install OCKD.
    cd ../bin
    sh ock_install.sh
    source /root/.bashrc 
  4. Set the local path to the path of the mf.conf configuration file. For details about how to edit the configuration file, see mf.conf.
    1. Go to the /opt/ock/conf directory.
      cd /opt/ock/conf
    2. Open the mf.conf file.
      vim mf.conf
    3. Press i to enter the insert mode and modify the data directories:
      ock.mf.ip_mask = 192.168.xx.xx-192.168.xx.xx # The IP address range contains the IP addresses of all nodes.
      ock.mf.protocol = tcp
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false
    4. Delete ock.ucache.rpc.tls.crl.path and ock.ucache.rpc.tls.sdk.crl.path.
    5. Press Esc, type :wq!, and press Enter to save the file and exit.
  5. Set the local path to the path of the ock.conf configuration file. For details about how to edit the configuration file, see ock.conf.
    1. Open the file.
      vim ock.conf
    2. Press i to enter the insert mode and modify the data directories:
      ock.ucache.rpc.transport.protocol = tcp
      ock.ucache.meta.node_lists = xx
      ock.zookeeper.server.url = xx:2181 # xx indicates the IP address of the ZooKeeper node.
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false
      ock.zookeeper.security.enable = false
      ock.ucache.rpc.auth.meta.principle.mapping = xx:master # xx indicates the IP address and name of the management node.
      ock.ucache.server.swap.path = /data/data1/ock_swap,/data/data2/ock_swap,/data/data3/ock_swap,/data/data4/ock_swap,/data/data5/ock_swap,/data/data6/ock_swap,/data/data7/ock_swap,/data/data8/ock_swap,/data/data9/ock_swap,/data/data10/ock_swap,/data/data11/ock_swap,/data/data12/ock_swap # Set it based on the actual of drives.
      ock.ucache.rpc.openssl.path=/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libssl.so
      ock.ucache.rpc.crypto.path = /opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libcrypto.so
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
      • The /home directory can be customized.
      • ock.ucache.meta.node_lists is the IP address of the management node.
      • ock.zookeeper.server.url is IP address of the ZooKeeper node.
        The following is an example for single-node ZooKeeper configuration:
        ock.zookeeper.server.url = 192.168.1.85:2181

        The following is an example for ZooKeeper cluster configuration:

        ock.zookeeper.server.url = 192.168.1.85:2181,192.168.1.86:2181,192.168.1.87:2181