Installing the Software

Before installing OmniShuffle, obtain the software installation package and verify its integrity. For details, see the section of environment requirements. Install OmniShuffle on all nodes and configure the parameter files.

Perform the following operations on all nodes.

1. Go to the nodes where you want to install OmniShuffle, and download the OmniShuffle software package from Obtaining Software to the /home directory.

   tar --no-same-owner -zxvf /home/ockadmin/BoostKit-omnishuffle_1.3.0.tar.gz
   cd /home/ockadmin/BoostKit-omnishuffle_1.3.0
   tar --no-same-owner -zxvf omnishuffleUtils_openeuler-aarch64.tar.gz

2. Modify parameters in the common_params.sh file.
   1. Go to the omnishuffleUtils/common directory.

      cd omnishuffleUtils/common

   2. Open the common_params.sh file.

      vi common_params.sh

   3. Press i to enter the insert mode and modify the following content of the file:

      ock_root_path=/home/ockadmin/opt
      spark_version="spark-3.1"

   4. Press Esc, type :wq!, and press Enter to save the file and exit.
3. Go to the bin directory and install OmniShuffle.

   cd ../bin
   sh ock_install.sh
   source ~/.bashrc 

4. Set the local path to the path of the mf.conf configuration file. For details about how to edit the configuration file, see mf.conf.
   1. Go to the /home/ockadmin/opt/ock/conf directory.

      cd /home/ockadmin/opt/ock/conf

   2. Open the mf.conf file.

      vi mf.conf

   3. Press i to enter the insert mode and modify the data directories:

      ock.mf.ip_mask = 192.168.xx.xx-192.168.xx.xx # The IP address range contains the IP addresses of all nodes.
      ock.mf.protocol = tcp
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false

   4. Delete ock.ucache.rpc.tls.crl.path and ock.ucache.rpc.tls.sdk.crl.path.
   5. Press Esc, type :wq!, and press Enter to save the file and exit.
5. Set the local path to the path of the ock.conf configuration file. For details about how to edit the configuration file, see ock.conf.
   1. Open the file.

      vi ock.conf

   2. Press i to enter the insert mode and modify the data directories:

      ock.ucache.rpc.transport.protocol = tcp
      ock.ucache.meta.node_lists = xx
      ock.zookeeper.server.url = xx:2181 # xx indicates the IP address of the ZooKeeper node.
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false
      ock.zookeeper.security.enable = false
      ock.ucache.rpc.auth.meta.principle.mapping = xx:master # xx indicates the IP address and name of the management node.
      ock.ucache.server.swap.path = /data/data1/ock_swap,/data/data2/ock_swap,/data/data3/ock_swap,/data/data4/ock_swap,/data/data5/ock_swap,/data/data6/ock_swap,/data/data7/ock_swap,/data/data8/ock_swap,/data/data9/ock_swap,/data/data10/ock_swap,/data/data11/ock_swap,/data/data12/ock_swap # Set it based on the actual of drives.
      ock.ucache.rpc.openssl.path=/home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libssl.so
      ock.ucache.rpc.crypto.path = /home/ockadmin/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libcrypto.so

   3. Press Esc, type :wq!, and press Enter to save the file and exit.
      

      • The /home/ockadmin directory can be customized.
      • ock.ucache.meta.node_lists is the IP address of the management node.
      • ock.zookeeper.server.url is IP address of the ZooKeeper node.
        The following is an example for single-node ZooKeeper configuration:

        ock.zookeeper.server.url = 192.168.1.100:2181

        The following is an example for ZooKeeper cluster configuration:

        ock.zookeeper.server.url = 192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181