Installing OmniShuffle
Perform the following operations on all nodes.
- Go to the nodes where you want to install OmniShuffle, and download the OmniShuffle software package from Table 3 to the /home directory.
tar -zxvf /home/BoostKit-omnishuffle_1.3.0.tar.gz cd /home/BoostKit-omnishuffle_1.3.0 tar -zxvf omnishuffleUtils_openeuler-aarch64.tar.gz
- Modify parameters in the common_params.sh file.
- Go to the omnishuffleUtils/common directory.
cd omnishuffleUtils/common
- Open the common_params.sh file.
vim common_params.sh
- Press i to enter the insert mode and modify the following content of the file:
ock_root_path=/opt spark_version="spark-3.1"
- Press Esc, type :wq!, and press Enter to save the file and exit.
- Go to the omnishuffleUtils/common directory.
- Go to the bin directory and install OCKD.
cd ../bin sh ock_install.sh source /root/.bashrc
- Set the local path to the path of the mf.conf configuration file. For details about how to edit the configuration file, see mf.conf.
- Go to the /opt/ock/conf directory.
cd /opt/ock/conf
- Open the mf.conf file.
vim mf.conf
- Press i to enter the insert mode and modify the data directories:
ock.mf.ip_mask = 192.168.xx.xx-192.168.xx.xx # The IP address range contains the IP addresses of all nodes. ock.mf.protocol = tcp ock.ucache.rpc.enableAuthentication = false ock.ucache.rpc.enableTLS = false ock.ucache.rpc.enableAuthorization = false
- Delete ock.ucache.rpc.tls.crl.path and ock.ucache.rpc.tls.sdk.crl.path.
- Press Esc, type :wq!, and press Enter to save the file and exit.
- Go to the /opt/ock/conf directory.
- Set the local path to the path of the ock.conf configuration file. For details about how to edit the configuration file, see ock.conf.
- Open the file.
vim ock.conf
- Press i to enter the insert mode and modify the data directories:
ock.ucache.rpc.transport.protocol = tcp ock.ucache.meta.node_lists = xx ock.zookeeper.server.url = xx:2181 # xx indicates the IP address of the ZooKeeper node. ock.ucache.rpc.enableAuthentication = false ock.ucache.rpc.enableTLS = false ock.ucache.rpc.enableAuthorization = false ock.zookeeper.security.enable = false ock.ucache.rpc.auth.meta.principle.mapping = xx:master # xx indicates the IP address and name of the management node. ock.ucache.server.swap.path = /data/data1/ock_swap,/data/data2/ock_swap,/data/data3/ock_swap,/data/data4/ock_swap,/data/data5/ock_swap,/data/data6/ock_swap,/data/data7/ock_swap,/data/data8/ock_swap,/data/data9/ock_swap,/data/data10/ock_swap,/data/data11/ock_swap,/data/data12/ock_swap # Set it based on the actual of drives. ock.ucache.rpc.openssl.path=/opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libssl.so ock.ucache.rpc.crypto.path = /opt/ock/ucache/23.0.0/linux-aarch64/lib/common/openssl/libcrypto.so
- Press Esc, type :wq!, and press Enter to save the file and exit.
- The /home directory can be customized.
- ock.ucache.meta.node_lists is the IP address of the management node.
- ock.zookeeper.server.url is IP address of the ZooKeeper node.The following is an example for single-node ZooKeeper configuration:
ock.zookeeper.server.url = 192.168.1.85:2181
The following is an example for ZooKeeper cluster configuration:
ock.zookeeper.server.url = 192.168.1.85:2181,192.168.1.86:2181,192.168.1.87:2181
- Open the file.
Parent topic: Local Installation