Installing the Software

Before installing OmniShuffle, obtain the software package and verify its integrity. For details, see the section of environment requirements. Install OmniShuffle on all nodes and configure the parameter files.

• Perform the following operations on all nodes.
• The value of spark_version in 2 must be the same as the actual version.

1. Go to the target nodes one by one. On each node, obtain the OmniShuffle software package from Obtaining the Software Package and save it to the /home/ockadmin directory, and run the following commands to decompress the package:

   cd /home/ockadmin
   tar --no-same-owner -zxvf /home/ockadmin/BoostKit-omnishuffle_1.7.0.tar.gz
   cd /home/ockadmin/BoostKit-omnishuffle_1.7.0
   tar --no-same-owner -zxvf omnishuffleUtils_aarch64.tar.gz

2. Modify parameters in the common_params.sh file.
   1. Go to the omniShuffleUtils/common directory.

      cd omnishuffleUtils/common

   2. Open the common_params.sh file.

      vi common_params.sh

   3. Press i to enter the insert mode and modify the following content of the file:

      ock_root_path=/home/ockadmin/opt
      spark_version="spark-3.3"
      ...
      # RSS or ESS: Configure the RSS or ESS mode.
      ock_mode="rss"
      ...

   4. Press Esc, type :wq!, and press Enter to save the file and exit.
3. Go to the bin directory and install OmniShuffle. Add the OCK_HOME, OCK_VERSION, and OCK_BINARY_TYPE environment variables by following instructions in 3.

   cd ../bin
   sh ock_install.sh
   source ~/.bashrc 

4. Delete unnecessary JAR packages from the installation directory based on the deployment mode and Spark version.
   • In RSS mode: Go to /home/ockadmin/opt/ock/jars and check that all JAR files are consistent with the following:

     ock-broadcast-sdk-24.0.0.jar
     ock-launch-cluster-24.0.0.jar
     ock-Shuffle-manager-24.0.0-for-spark-3.3.jar # 3.3, which is the same as the Spark version.
     ock-Shuffle-sdk-24.0.0.jar
     zk-server-auth-plugin-tls-24.0.0-assembly.jar
     zk-server-auth-plugin-keytab-24.0.0-assembly.jar

   • In ESS mode: Go to /home/ockadmin/opt/ock/jars and check that all JAR files are consistent with the following:

     ock-broadcast-sdk-24.0.0.jar
     ock-launch-cluster-24.0.0.jar
     ock-Shuffle-manager-24.0.0-for-spark-3.3.jar # 3.3, which is the same as the Spark version.
     ock-Shuffle-sdk-24.0.0.jar
     zk-server-auth-plugin-tls-24.0.0-assembly.jar
     zk-server-auth-plugin-keytab-24.0.0-assembly.jar

   

   The JAR package user is ockadmin. Ensure that the user who submits the Spark task has the permission to modify the JAR package. Set the permission to 550 or higher.

5. Set the local path to the path of the mf.conf configuration file. For details about how to edit the configuration file, see mf.conf.
   1. Go to the /home/ockadmin/opt/ock/conf directory.

      cd /home/ockadmin/opt/ock/conf

   2. Open the mf.conf file.

      vi mf.conf

   3. Press i to enter the insert mode and modify the data directories:

      ock.mf.ip_mask = 192.168.xx.xx-192.168.xx.xx # The IP address range contains the IP addresses of all nodes.
      ock.mf.protocol = tcp
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false

   4. Delete ock.ucache.rpc.tls.crl.path and ock.ucache.rpc.tls.sdk.crl.path.
   5. Press Esc, type :wq!, and press Enter to save the file and exit.
6. Set the local path to the path of the ock.conf configuration file. For details about how to edit the configuration file, see ock.conf.
   1. Open the ock.conf file.

      vi ock.conf

   2. Press i to enter the insert mode and modify the data directories:

      ock.ucache.rpc.transport.protocol = tcp
      ock.ucache.rpc.shuffle_server.port = 3891
      ock.ucache.master.ip = 192.168.xx.xx
      
      ock.ucache.rpc.enableAuthentication = false
      ock.ucache.rpc.enableTLS = false
      ock.ucache.rpc.enableAuthorization = false
      ock.zookeeper.security.enable = false
      
      ock.ucache.server.swap.threshold.higher_watermark = 60
      ock.ucache.server.swap.threshold.lower_watermark = 20
      ock.ucache.server.swap.path = /data/data1/ock_swap,/data/data2/ock_swap,/data/data3/ock_swap,/data/data4/ock_swap // Set the parameters based on the actual number of drives. Mount NVMe drives in RSS mode.
      ock.ucache.rpc.openssl.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl/libssl.so
      ock.ucache.rpc.crypto.path = /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl/libcrypto.so
      
      ock.ucache.rss.mode = false  // Set it to false in ESS mode and to true in RSS mode.
      ock.ucache.rpc.transport.ipfilter = 192.168.xx.xx/24

   3. Press Esc, type :wq!, and press Enter to save the file and exit.
      

      • The /home/ockadmin directory can be customized.
      • ock.ucache.rpc.transport.ipfilter indicates an IP address filter. For example, if the IP address of a cluster node is 192.168.1.x, set this parameter to 192.168.0.1/24.