Installing OmniShuffle Shuffle

Before installing OmniShuffle, obtain the software package and verify its integrity. For details, see the section of environment requirements. Install OmniShuffle on all nodes and configure the parameter files.

Perform the installation operations on all nodes.

Go to the target nodes one by one. On each node, obtain the OmniShuffle software package and save it to the /home/ockadmin directory, and run the following commands to decompress the package:

cd /home/ockadmin
tar --no-same-owner -zxvf /home/ockadmin/BoostKit-omnishuffle_1.9.0.tar.gz
cd /home/ockadmin/BoostKit-omnishuffle_1.9.0
tar --no-same-owner -zxvf omnishuffleUtils_aarch64.tar.gz

Modify parameters in the common_params.sh file.
1. Go to the omniShuffleUtils/common directory.
```
cd omnishuffleUtils/common
```
2. Open the common_params.sh file.
```
vi common_params.sh
```
3. Press i to enter the insert mode and modify the following content of the file:
```
ock_root_path=/home/ockadmin/opt
spark_version="spark-3.3"
...
# RSS or ESS: Configure the RSS or ESS mode.
ock_mode="rss"
...
```
  spark_version: Spark version. Replace it with the actual one.
4. Press Esc, type :wq!, and press Enter to save the file and exit.
Go to the bin directory and install OmniShuffle. Add the OCK_HOME, OCK_VERSION, and OCK_BINARY_TYPE environment variables by following instructions in 3.
```
cd ../bin
sh ock_install.sh
source ~/.bashrc 
```

Delete unnecessary JAR packages from the installation directory based on the deployment mode and Spark version.

RSS mode

Go to /home/ockadmin/opt/ock/jars and check that all JAR files are consistent with the following:

ock-broadcast-sdk-24.0.0.jar
ock-launch-cluster-24.0.0.jar
ock-Shuffle-manager-24.0.0-for-spark-3.3.jar # 3.3, which is the same as the Spark version.
ock-Shuffle-sdk-24.0.0.jar
zk-server-auth-plugin-tls-24.0.0-assembly.jar
zk-server-auth-plugin-keytab-24.0.0-assembly.jar

ESS mode

Go to /home/ockadmin/opt/ock/jars and check that all JAR files are consistent with the following:

ock-broadcast-sdk-24.0.0.jar
ock-launch-cluster-24.0.0.jar
ock-Shuffle-manager-24.0.0-for-spark-3.3.jar # 3.3, which is the same as the Spark version.
ock-Shuffle-sdk-24.0.0.jar
zk-server-auth-plugin-tls-24.0.0-assembly.jar
zk-server-auth-plugin-keytab-24.0.0-assembly.jar

The JAR package user is ockadmin. Ensure that the user who submits the Spark task has the permission to modify the JAR package. Set the permission to 550 or higher.

Set the local path to the path of the mf.conf configuration file. For details about how to edit the configuration file, see mf.conf.
1. Go to the /home/ockadmin/opt/ock/conf directory.
```
cd /home/ockadmin/opt/ock/conf
```
2. Open the mf.conf file.
```
vi mf.conf
```
3. Press i to enter the insert mode and modify the data directories:
```
ock.mf.ip_mask = 192.168.xx.xx-192.168.xx.xx # The IP address range contains the IP addresses of all nodes.
ock.mf.protocol = tcp
ock.ucache.rpc.enableAuthentication = false
ock.ucache.rpc.enableTLS = false
ock.ucache.rpc.enableAuthorization = false
```
4. Delete ock.ucache.rpc.tls.crl.path and ock.ucache.rpc.tls.sdk.crl.path.
5. Press Esc, type :wq!, and press Enter to save the file and exit.

Set the local path to the path of the ock.conf configuration file. For details about how to edit the configuration file, see ock.conf.

Open the ock.conf file.
```
vi ock.conf
```

Press i to enter the insert mode and modify the data directories:

ock.ucache.rpc.transport.protocol = tcp
ock.ucache.rpc.shuffle_server.port = 3891
ock.ucache.master.ip = 192.168.xx.xx

ock.ucache.rpc.enableAuthentication = false
ock.ucache.rpc.enableTLS = false
ock.ucache.rpc.enableAuthorization = false
ock.zookeeper.security.enable = false

ock.ucache.server.swap.threshold.higher_watermark = 60
ock.ucache.server.swap.threshold.lower_watermark = 20
ock.ucache.server.swap.path = /data/data1/ock_swap,/data/data2/ock_swap,/data/data3/ock_swap,/data/data4/ock_swap // Set the parameters based on the actual number of drives. Mount NVMe drives in RSS mode.
ock.ucache.rpc.openssl.path=/home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl/libssl.so
ock.ucache.rpc.crypto.path = /home/ockadmin/opt/ock/ucache/24.0.0/linux-aarch64/lib/common/openssl/libcrypto.so

ock.ucache.rss.mode = false  // Set it to false in ESS mode and to true in RSS mode.
ock.ucache.rpc.transport.ipfilter = 192.168.xx.xx/24

The /home/ockadmin path can be customized.
ock.ucache.rpc.transport.ipfilter indicates an IP address filter. For example, if the IP address of a cluster node is 192.168.1.x, set this parameter to 192.168.1.0/24.

Press Esc, type :wq!, and press Enter to save the file and exit.

Verify the installation. For details, see Installing the Feature.
If the startup is successful, the feature is successfully installed.

Parent topic: Installing the Feature