RDMA Network Acceleration Feature Guide

Feature Description

Overview

Unified Communication X (UCX) is a general communication framework between the application layer and the driver layer. It provides unified communication interfaces for applications and supports TCP/IP and Remote Direct Memory Access (RDMA). UCX was designed for high performance compact scenarios where RDMA can be enabled to achieve high throughput and low latency. This document describes how to deploy Ceph and configure UCX on a Kunpeng server running openEuler 20.03.

With the development of software-defined storage, the requirements for latency and CPU usage are increasingly strict, and RDMA becomes a preferred choice. The Kunpeng BoostKit for SDS RDMA network acceleration feature applies the UCX framework to enable RDMA in front- and back-end networks of a Ceph cluster. UCX can enable RDMA and TCP/IP together. It has been applied to both the front-end network and cluster network.

Other Information

Before configuring this feature, learn about the license requirements, constraints, and principles.

Availability

Software versions: Ceph 14.2.8 and UCX 1.14.1

Constraints

The RDMA network acceleration feature is implemented based on UCX + Ceph 14.2.8. UCX + other distributed storage modes are not supported.

Principles

Open source Ceph distributed storage has the following communication frameworks:

Simple: basic client-server model. Two threads are created for each connection to transmit and receive messages. The number of threads increases as the number of connections increases.
Async: asynchronous communication framework. Message transmission and receipt are processed by an asynchronous message framework, and the processing results are asynchronously sent to application threads. In this mode, the number of threads does not increase as the number of connections increases, but the message transmission and receipt performance deteriorates due to asynchronous waiting. This framework is widely used.
XIO: a new communication framework. It has not been put into commercial use.

The RDMA network acceleration feature applies UCX to the asynchronous framework used by open source Ceph distributed storage to enable end-to-end RDMA.

Before configuring this feature, learn about the license requirements, constraints, and principles.

Environment Requirements

This document provides guidance based on the Kunpeng server and openEuler OS. Before performing operations, ensure that your hardware and software meet the requirements.

Environment Networking

Ceph is used in the environment, including three client nodes and three server nodes.

Hardware Requirements

Table 1 Hardware requirements

Item	Specifications
Server	TaiShan 200 with Kunpeng processors
CPU	Kunpeng 920
NIC	Client node: one 2 × 25GE NIC, 50GE in total Ceph node: one 4 × 25GE NIC or two 2 × 25GE NICs, 100GE in total

OS and Software Requirements

Table 2 OS and software requirements

Item	Version	How to Obtain
Physical machine OS	openEuler 20.03 LTS SP4	OS
Ceph	14.2.8	Ceph
UCX	1.14.1	UCX

Obtaining the Software Package

Before compiling and deploying UCX, prepare the following software package.

Software Package	Description	How to Obtain
ceph-14.2.8-ucx.patch	Patch for adapting Ceph to UCX	Patch

Compiling and Installing the UCX and Ceph Packages

Compiling and Installing UCX Packages

Compile and deploy UCX open-source software packages, including compiling and generating UCX RPM packages required for compiling Ceph.

Obtain the UCX open-source software packages.

wget https://github.com/openucx/ucx/releases/download/v1.14.1/ucx-1.14.1-1.el7.src.rpm --no-check-certificate

For details about how to obtain the packages, see Table 2.

Set the directory for building RPM packages.
1. Open the /root/.rpmmacros file.
```
vi /root/.rpmmacros
```
2. Press i to enter the insert mode, set %_topdir to the RPM package build directory (/root/rpmbuild for example), and comment out other lines.
```
%_topdir /root/rpmbuild
```
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
4. Create a build directory in the rpmbuild directory.
```
yum install rpmdevtools
rpmdev-setuptree
```
Install the UCX RPM packages.
```
rpm -ivh ucx-1.14.1-1.el7.src.rpm
```

Install the compilation dependencies.

yum install libibverbs-devel librdmacm-devel libtool numactl-devel

Compile and build RPM packages. In the rpmbuild directory, compile and build the ucx.spec file to generate RPM packages.
```
cd /root/rpmbuild/SPECS
rpmbuild -bb ucx.spec
```
After the build is complete, eight RPM packages are generated in the /root/rpmbuild/RPMS/aarch64 directory, as shown in the following figure.

Install the RPM packages.

cd /root/rpmbuild/RPMS/aarch64

rpm -ivh ucx-1.14.1-1.aarch64.rpm
rpm -ivh ucx-cma-1.14.1-1.aarch64.rpm
rpm -ivh ucx-debuginfo-1.14.1-1.aarch64.rpm
rpm -ivh ucx-debugsource-1.14.1-1.aarch64.rpm
rpm -ivh ucx-devel-1.14.1-1.aarch64.rpm
rpm -ivh ucx-ib-1.14.1-1.aarch64.rpm
rpm -ivh ucx-rdmacm-1.14.1-1.aarch64.rpm
rpm -ivh ucx-static-1.14.1-1.aarch64.rpm

Compiling Ceph Packages

Installing Dependencies

Install common components.

yum install CUnit-devel boost-random checkpolicy cmake cryptsetup-devel expat-devel fmt-devel fuse-devel gperf java-devel junit keyutils-libs-devel libaio-devel libbabeltrace-devel libblkid-devel libcap-ng-devel libcurl-devel numactl-devel libicu-devel libnl3-devel liboath-devel librabbitmq-devel librdkafka-devel librdmacm-devel libtool libxml2-devel lttng-ust-devel lua-devel lz4-devel make nasm ncurses-devel ninja-build nss-devel openldap-devel openssl-devel libudev-devel python3-Cython python3-devel python3-prettytable python3-pyyaml python3-setuptools python3-sphinx re2-devel selinux-policy-devel sharutils snappy-devel sqlite-devel sudo thrift-devel valgrind-devel xfsprogs-devel xmlstarlet doxygen python2-Cython createrepo gperftools leveldb-devel yasm -y

Install RDMA dependencies.

yum install libibverbs-devel rdma-core-devel numactl-devel -y

Compiling Ceph

Download the ceph-14.2.8 source code.

wget https://download.ceph.com/tarballs/ceph-14.2.8.tar.gz --no-check-certificate

Apply the UCX patch by placing ceph-14.2.8-ucx.patch in the current directory.

tar -zxvf ceph-14.2.8.tar.gz
cd ceph-14.2.8
patch -p1 < ceph-14.2.8-ucx.patch

Compile Ceph.

cd ..
tar -zcvf ceph-14.2.8.tar.bz2 ceph-14.2.8
cp ceph-14.2.8/ceph.spec /root/rpmbuild/SPECS/
cp ceph-14.2.8.tar.bz2 /root/rpmbuild/SOURCES/
rpmbuild -bb /root/rpmbuild/SPECS/ceph.spec

Deploying Ceph

Configuring the Deployment Environment

Disable the firewall.

To disable the firewall on the current node, run the following commands on all server and client nodes:
```
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
```
Configure host names.

Configure the permanent static host names. Set the names of the server nodes to ceph1, ceph2, and ceph3 and the names of the client nodes to client1, client2, and client3.
1. Configure node names.
  1. Set the host name of server node 1 to ceph1.
```
hostnamectl --static set-hostname ceph1
```
  2. Set the host name of client node 1 to client1.
```
hostnamectl --static set-hostname client1 
```
    Configure host names for other nodes by referring to the preceding examples.
2. Modify the domain name resolution file.
```
vi /etc/hosts
```
  Add the following content to the /etc/hosts file on all server and client nodes:
```
192.168.3.166   ceph1 
192.168.3.167   ceph2
192.168.3.168   ceph3
192.168.3.160   client1 
192.168.3.161   client2 
192.168.3.162   client3
```
  - The preceding IP addresses are only examples. Replace them as required. You can run the ip a command to obtain the actual IP addresses.
  - You are advised to set the host names of the server nodes to ceph1, ceph2, and ceph3.
  - You are advised to set the host names of the client nodes to client1, client2, and client3.
  - The preceding example uses three servers and three clients. Change the number of nodes based on the site requirements.
Configure NTP.

Ceph automatically checks the time between storage nodes. If the time difference between different nodes is large, an alarm will be generated. You need to configure clock synchronization between nodes.
1. Install NTP.
  1. Install NTP on all server and client nodes.
```
yum -y install ntp ntpdate
```
  2. Back up the original configurations of all server and client nodes.
```
cd /etc && mv ntp.conf ntp.conf.bak
```
  3. Create an NTP file on ceph1, which serves as the NTP server.
```
vi /etc/ntp.conf
```
    Add the following NTP server configuration to the NTP file:
```
restrict 127.0.0.1 
restrict ::1 
restrict 192.168.3.0 mask 255.255.255.0
server 127.127.1.0 
fudge 127.127.1.0 
stratum 8
restrict default kod nomodify notrap nopeer noquery
interface ignore wildcard
interface listen x.x.x.x (*IP address of the server*)
```
    restrict 192.168.3.0 mask 255.255.255.0 indicates the network segment and subnet mask of ceph1.
  4. Create an NTP file on ceph2, ceph3, and all client nodes.
```
vi /etc/ntp.conf
```
    Add the following content, in which the IP address is the IP address of ceph1:
```
server 192.168.3.166 
```
2. Start NTP.
  1. Start NTP on ceph1 and check its status.
```
systemctl start ntpd 
systemctl enable ntpd 
systemctl status ntpd 
```
  2. Forcibly synchronize the NTP server (ceph1) time to all the other nodes.
```
ntpdate ceph1 
```
  3. Write the hardware clock to all nodes except ceph1 to prevent configuration failures after the restart.
```
hwclock -w 
```
  4. Install and start the crontab tool on all nodes except ceph1.
```
yum install -y crontabs
systemctl enable crond.service
systemctl start crond 
crontab -e 
```
  5. Add the following content so that all the other nodes automatically synchronize time with ceph1 every 10 minutes:
```
*/10 * * * * /usr/sbin/ntpdate 192.168.3.166
```
Configure password-free login.

Generate a public key on ceph1 and deliver it to other server and client nodes.
- For a three-node network of ceph1, ceph2, ceph3, client1, client2, and client3:
```
ssh-keygen -t rsa 
for i in {1..3};do ssh-copy-id ceph$i;done
for i in {1..3};do ssh-copy-id client$i;done
```
- For a network of ceph1 and client1:
```
ssh-keygen -t rsa
ssh-copy-id ceph1
ssh-copy-id client1
```
After entering the first command ssh-keygen -t rsa, press Enter to use the default configuration.
Set the permissive mode.

Set the permissive mode on all server and client nodes.
- Temporarily disable the SELinux function. The setting becomes invalid after the OS is restarted.
```
setenforce permissive
```
- Set the permissive mode permanently. The configuration takes effect upon the next restart.
```
vi /etc/selinux/config
```
  Set SELINUX to permissive.

Installing Ceph

Install Ceph on all server and client nodes.

Set the Yum certificate verification status of all server nodes and client nodes to verification-free.
1. Open the file.
```
vi /etc/yum.conf
```
2. Press i to enter the insert mode and add the following content to the end of the file:
```
sslverify=false
deltarpm=0
```
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.

(Optional) Configure the local source of gperftools on all server and client nodes.

Download the RPM packages of gperftools-devel-2.7.7 and gperftools-libs-2.7.7.

mkdir -p /home/gperftools-2.7-7 && cd /home/gperftools-2.7-7
wget --no-check-certificate  https://repo.openeuler.org/openEuler-20.03-LTS/OS/aarch64/Packages/gperftools-devel-2.7-7.oe1.aarch64.rpm
wget --no-check-certificate  https://repo.openeuler.org/openEuler-20.03-LTS/OS/aarch64/Packages/gperftools-libs-2.7-7.oe1.aarch64.rpm
createrepo .

Open the /etc/yum.repos.d/openEuler.repo file.
```
vi /etc/yum.repos.d/openEuler.repo
```

Press i to enter the insert mode and add the following content to the end of the file:

[gperftools-2.7-7]
name=gperftools-2.7-7
baseurl=file:///home/gperftools-2.7-7
enabled=1
gpgcheck=0
priority=1

Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.

Install the RDMA dependencies on all server and client nodes.

yum install libibverbs-devel rdma-core-devel numactl-devel -y

Install the UCX RPM packages on all server and client nodes. (Place the RPM packages generated in 4.1 on each node.)

rpm -ivh ucx-1.14.1-1.aarch64.rpm
rpm -ivh ucx-cma-1.14.1-1.aarch64.rpm
rpm -ivh ucx-debuginfo-1.14.1-1.aarch64.rpm
rpm -ivh ucx-debugsource-1.14.1-1.aarch64.rpm
rpm -ivh ucx-devel-1.14.1-1.aarch64.rpm
rpm -ivh ucx-ib-1.14.1-1.aarch64.rpm
rpm -ivh ucx-rdmacm-1.14.1-1.aarch64.rpm
rpm -ivh ucx-static-1.14.1-1.aarch64.rpm

Save the Ceph RPM packages with UCX enabled (Compiling Ceph) to /home/ceph-ucx and configure the local repository.
```
vi /etc/yum.repos.d/local.repo
```
Add the following content to the end of the file, save the file, and exit:
```
[ceph-ucx]
name=ceph-ucx
baseurl=file:///home/ceph-ucx
enabled=1
gpgcheck=0
priority=1
```
Update the Yum repository.
```
cd /home/ceph-ucx
createrepo .
```
Install Ceph on all server and client nodes.
```
dnf -y install librados2-14.2.8 ceph-14.2.8
pip install prettytable werkzeug
```
If Ceph fails to be installed, configure a network proxy.

During Ceph installation, verify that the version of gperftools is 2.7-7.
Install ceph-deploy on ceph1.
```
pip install ceph-deploy
```
Adapt to openEuler.
1. Open the /lib/python2.7/site-packages/ceph_deploy/hosts/__init__.py file on ceph1.
```
vi /lib/python2.7/site-packages/ceph_deploy/hosts/__init__.py
```
2. Press i to enter the insert mode and add the following code to the _get_distro function:
```
'openeuler':fedora,
```
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
View the Ceph version on each node.
```
ceph -v
```
If the following information is displayed, Ceph is installed.
```
ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)
```

Install Ceph on all server and client nodes.

Deploying Ceph

Deploying MON

The Monitor (MON) monitors the status of the Ceph cluster. You only need to deploy MON on the primary node (ceph1 is used as an example).

Create a Ceph cluster. (ceph1, ceph2, and ceph3 are used as examples.)
```
cd /etc/ceph
ceph-deploy new ceph1 ceph2 ceph3
```
Configure global parameters and MON parameters for the Ceph cluster.

Configuring nodes and using ceph-deploy to configure OSDs need to be performed in the /etc/ceph directory. Otherwise, an error may occur.
1. Open the ceph.conf file that is automatically generated in the /etc/ceph directory.
```
vi /etc/ceph/ceph.conf
```
2. Press i to enter the insert mode and change the content in ceph.conf to the following information (use the latest fsid):
```
[global]
fsid = f5a4f55c-d25b-4339-a1ab-0fceb4a2996f
mon_initial_members = ceph1, ceph2, ceph3
mon_host = 192.168.3.166,192.168.3.167,192.168.3.168
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public_network = 192.168.65.0/24
cluster_network = 192.168.66.0/24

[mon]
mon_allow_pool_delete = true
```
  For a single-node environment, add the following content below [global]:
```
osd_pool_default_size = 1
osd_pool_default_min_size = 1
```
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
In Ceph 14.2.8, when the BlueStore engine is used, the buffer of the BlueFS is enabled by default. As a result, the system memory may be fully occupied by the buffer or cache, causing performance deterioration. You can use either of the following methods to solve the problem:
- If the cluster load is not heavy, set bluefs_buffered_io to false.
- Periodically run the following command to forcibly reclaim the memory occupied by the buffer or cache:
```
echo 3 > /proc/sys/vm/drop_caches
```
Initialize MONs and collect keys.
```
ceph-deploy mon create-initial
```

Copy ceph.client.admin.keyring generated in 3 to each node.

ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3 client1 client2 client3

Check the Ceph cluster status to determine whether the MONs are configured.

ceph -s

Expected result of successful configuration:

cluster:
id:     f6b3c38c-7241-44b3-b433-52e276dd53c6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 25h)

If MONs fails to be generated, the permission configuration may be incorrect. In this case, you need to configure the Ceph group and users.

/usr/sbin/groupadd ceph -g 167 -o -r 2>/dev/null || :/usr/sbin/useradd ceph -u 167 -o -r -g ceph -s /sbin/nologin -c "Ceph daemons" -d /var/lib/ceph 2>/dev/null || :

Deploying MGR

The Manager (MGR) is a key component for Ceph cluster management. It collects the status and runtime metrics of a Ceph cluster. You only need to deploy MGRs on ceph1, and then synchronize the configuration to ceph2 and ceph3.

Deploy MGRs.

ceph-deploy mgr create ceph1 ceph2 ceph3

Check the Ceph cluster status to determine whether the MGRs are deployed.

ceph -s

The deployment is successful if the command output is similar to the following:

cluster:
id:     f6b3c38c-7241-44b3-b433-52e276dd53c6
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 25h)
mgr: ceph1(active, since 2d), standbys: ceph2, ceph3

Adding OSDs

In this example, the NVMe SSD is divided into 12 60-GB partitions as WAL partitions and 12 180-GB partitions as DB partitions.

Create a partition.sh script. (Skip this step if partitioning is not required.)
```
vi partition.sh
```

Add the following content to the script: (Assume that a single NVMe SSD is divided into 12 partitions. Skip this step if partitioning is not required.)

#!/bin/bash

parted /dev/nvme0n1 mklabel gpt

for j in `seq 1 12`
do
((b = $(( $j * 8 ))))
((a = $(( $b - 8 ))))
((c = $(( $b - 6 ))))
str="%"
echo $a
echo $b
echo $c
parted /dev/nvme0n1 mkpart primary ${a}${str} ${c}${str}
parted /dev/nvme0n1 mkpart primary ${c}${str} ${b}${str}
done

Run the script. (Skip this step if partitioning is not required.)
```
bash partition.sh
```
Create a create_osd.sh script on ceph1 and deploy OSDs on the 12 partitions on each server.
```
vi /etc/ceph/create_osd.sh
```

Add the following content:

#!/bin/bash

for node in ceph1 ceph2 ceph3
do
for i in {0..7}
do
ceph-deploy osd create ${node} --data /dev/nvme${i}n1
done
done

Run the script on ceph1.
```
bash create_osd.sh
```
Check whether the created OSDs are normal.
```
ceph -s
```

In this example, the NVMe SSD is divided into 12 60-GB partitions as WAL partitions and 12 180-GB partitions as DB partitions.

Uninstalling Ceph

You can delete all Ceph components on nodes to uninstall Ceph.

Stop all Ceph service processes.
```
systemctl stop ceph.target
```
Uninstall Ceph.
```
yum rm ceph-14.2.8 librados2-14.2.8
```

Delete Ceph-related directories.

rm -rf /var/lib/ceph/*
rm -rf /etc/ceph/*
rm -rf /var/run/ceph/*

You can delete all Ceph components on nodes to uninstall Ceph.

Switching to the UCX Networking for a Cluster

Before enabling UCX, add UCX-related configurations and configure UCX environment variables in the Ceph configuration file. If the UCX multi-rail function is required, configure the UCX_MAX_RNDV_RAILS and UCX_MAX_EAGER_RAILS options.

Check whether the hardware and driver support RoCE. The Mellanox NIC is used as an example.
```
lspci | grep Mellanox
```
If RoCE is available, the following information will be displayed:
Stop Ceph on all server nodes.
```
systemctl stop ceph.target
```
Modify the Ceph configuration file on all server and client nodes. Add the following information under the global field in /etc/ceph/ceph.conf:
```
ms_type = async+ucx
ms_public_type = async+ucx
ms_cluster_type = async+ucx
ms_async_ucx_device=mlx5_0:1,mlx5_1:1
ms_async_ucx_tls=rc_verbs,self
ms_async_ucx_max_recv=14
```
- You can use show_gids to query device names and enter multiple network devices in ms_async_ucx_device. If the show_gids command is abnormal, see Updating the NIC Firmware and Driver.
- To enable UCX on the front-end network, set ms_public_type to async+ucx. To enable UCX only on the back-end network, set both ms_type and ms_public_type to async+posix.
- The IP addresses of cluster_network and public_network must be the same as those of the UCX devices (RoCE interfaces).
Configure UCX environment variables on all server and client nodes. Add the following information to the blank area in /etc/sysconfig/ceph:
```
UCX_MODULE_DIR=/lib64/ucx
UCX_RNDV_THRESH=32k
UCX_MEM_MMAP_HOOK_MODE=none
UCX_MAX_RNDV_RAILS=4
UCX_MAX_EAGER_RAILS=4
UCX_PROTO_ENABLE=y
```
- If UCX logs are required, add the following configurations:
```
UCX_LOG_FILE=/var/log/ceph/ucx_%p.log
UCX_LOG_LEVEL=DEBUG
```
- UCX_MEM_MMAP_HOOK_MODE can be set to reloc, bistro, or none. If the TCMalloc huge page is enabled, set it to reloc.
- If two interfaces need to be used at the same time, enable the UCX multi-rail function and set UCX_MAX_RNDV_RAILS and UCX_MAX_EAGER_RAILS to 2 or larger (value range: 1 to 4). The UCX multi-rail configuration can better balance traffic than the bond mode and achieve higher network bandwidth.
Modify the memory limit on all server and client nodes. Add the following information to the blank area in /etc/security/limits.conf:
```
root soft memlock unlimited
root hard memlock unlimited
ceph soft memlock unlimited
ceph hard memlock unlimited
```
Modify the Ceph configuration files in systemd. Add the following information to the service field in ceph-mds@.service, ceph-mgr@.service, ceph-mon@.service, and ceph-osd@.service of /lib/systemd/system/:
```
LimitMEMLOCK=infinity
LimitCORE=infinity
PrivateDevices=no
```
Add the following information to the After and Wants fields in ceph-mds@.service, ceph-mgr@.service, ceph-mon@.service, and ceph-osd@.service of /lib/systemd/system/:
Configure the openibd to wait for 60 seconds after startup.
```
vim /usr/lib/systemd/system/openibd.service
```
Configure the following configuration item:
```
ExecStartPost=/bin/sleep 60
```
Add the following configurations on all server and client nodes:
```
ulimit -l unlimited
ulimit -n 1048576
```
Before starting UCX, check whether all the UCX-related installation packages are installed.

Ensure that the four installation packages have been installed. Otherwise, the OSD service may exit.
```
rpm -qa | grep ucx
```
Expected result:
Update the configuration and start Ceph.
```
systemctl daemon-reload
systemctl start ceph.target
```
In heavy-load scenarios, only 256 images are supported.

Configuring Flow Control and Checking Traffic (RoCE Networking)

Configuring Switches

To configure a lossless network, you need to configure switches. This section uses the HUAWEI CE6863-48S6CQ switch as an example to describe how to execute a flow control policy.

Log in to the switch and enable priority-based flow control (PFC).
```
system-view
dcb pfc
priority 0
commit
quit
```
Check whether PFC is enabled.
```
display dcb pfc-profile
```
If 0 is returned, the configuration in 1 is successful.

Configure Explicit Congestion Notification (ECN).

system-view
drop-profile ecn
color green buffer-size low-limit 247520 high-limit 18000000 discard-percentage 100
commit
quit

Check the ECN profile.
```
display drop-profile ecn
```
If the following information is displayed, the configuration in 3 takes effect.

Configure flow control for each traffic interface. A 25GE interface 1/0/5 is used as an example.

interface 25GE 1/0/5
qos queue 0 wred ecn
qos queue 0 ecn
dcb pfc enable mode manual
dcb pfc buffer 0 xoff static 1500 cells
commit
quit

Check whether the configuration takes effect.
```
interface 25GE 1/0/5
display this
```
If the information in 5 is returned, the flow control is configured.

To configure a lossless network, you need to configure switches. This section uses the HUAWEI CE6863-48S6CQ switch as an example to describe how to execute a flow control policy.

Checking Interface Traffic

You can check whether the switch configuration takes effect on the service side by checking whether there is traffic on interfaces.

Configure queue priorities for RoCE NICs on all nodes.

Even if you have bonded two interfaces (mode 0/2/4), you still need to configure the priority for each interface to achieve optimal network performance.
```
mlnx_qos -i enp133s0f0 -f 1,0,0,0,0,0,0,0
mlnx_qos -i enp133s0f1 -f 1,0,0,0,0,0,0,0
```
Check whether there is traffic on the NIC.
```
watch -n 1 "ethtool -S enp133s0f0 | grep prio"
watch -n 1 "ethtool -S enp133s0f1 | grep prio"
```
If the returned traffic changes, there is traffic on the NIC.

Figure 1 Example command output for enp133s0f0

Figure 2 Example command output for enp133s0f1

You can check whether the switch configuration takes effect on the service side by checking whether there is traffic on interfaces.

FAQs

Updating the NIC Firmware and Driver

Download the firmware package and decompress it. (The CX-5 NIC is used as an example.)

Upgrade the firmware.

cd NIC-SP382-CX5-FW-16.32.1010-ARM
./install.sh upgrade

Install driver dependencies.

yum install createrepo perl pciutils gcc-gfortran tcsh expat glib2 tcl libstdc++ bc tk gtk2 atk cairo numactl pkgconfig ethtool lsof rpm-build libxml2-python python autoconf automake libtool

Download the NIC driver.
Install the driver.
1. Decompress the downloaded ISO file.
```
mkdir /mnt/iso
mount -o loop ***.iso /mnt/iso
cd /mnt/iso
```
  - ***.iso indicates the ISO file corresponding to the NIC driver, for example, onboard_driver_openEuler20.03.iso.
  - ***.tgz in the following commands indicates the driver package in the ISO file. Replace it with the actual name.
2. Install the driver in either of the following ways:
  - Method 1: Decompress the installation package to install the driver.
```
cp ***.tgz /home
cd /home
tar xf ***.tgz
cd ***
./mlnxofedinstall --force --without-depcheck --without-fw-update --add-kernel-support  --skip-distro-check
```
  - Method 2: Use the automatic installation script.
```
./install.sh  # (Refer to the readme_en.txt file in the same directory.)
```
Reload the driver.
```
dracut -f
/etc/init.d/openibd restart
```
If the driver is in use, stop related services so that the driver can be reloaded.
Reboot the node.
```
reboot
```

The recommended versions are as follows:

Firmware version: 16.32.1010 (HUA0000000024)
Driver version:
- openEuler 20.03 (Arm): 24.01-0.3.3
- openEuler 20.03 (x86): 5.8-1.1.2

How Can I Rectify a Driver Error Reported During Service Execution?

Symptom

When a service operation is performed, an error message is displayed, as shown in Figure 1 Error message.

Figure 1 Error message

Solution

If the preceding error information is displayed in dmesg, configure the following parameter on the node:

mst start
mlxconfig -d 85:00.0 -y s PF_LOG_BAR_SIZE=8
reboot

85:00.0 indicates the PCIe number of the NIC.

How Can I Update the SP670 Driver to Support UCX?

Download the latest firmware and driver installation packages (NIC-FW-17.12.1.2.tar.gz and SDK_LINUX-17.12.1.2-openEuler22.03SP4-aarch64.tar.gz) and decompress them. For details, see the website.

Upgrade the firmware.

tar -zvxf NIC-FW-17.12.1.2.tar.gz
cd NIC-FW-17.12.1.2
rpm -ivh tool/aarch64/hinicadm3-17.12.1.2-1.aarch64.rpm
hinicadm3 updatefw -i hinic0 -f ./SP670/Hinic3_flash.bin -or

Install the driver. You need to install the driver version corresponding to the OS kernel. The following uses openEuler 22.03 LTS SP4 as an example.

tar -zvxf SDK_LINUX-17.12.1.2-openEuler22.03SP4-aarch64.tar.gz
cd SDK_LINUX-17.12.1.2-openEuler22.03SP4-aarch64
rpm -ivh nic/hisdk3-17.12.1.2_5.10.0_216.0.0.115.oe2203sp4.aarch64-1.aarch64.rpm
rpm -ivh nic/hinic3-17.12.1.2_5.10.0_216.0.0.115.oe2203sp4.aarch64-1.aarch64.rpm
rpm -ivh roce/hiroce3-17.12.1.2_5.10.0_216.0.0.115.oe2203sp4.aarch64-1.aarch64.rpm

Run the reboot command to restart the server.

How Can I Rectify an Error Reported When UCX Uses SP670?

Symptom

The following error message is displayed when ucx_info -d scans devices:

Cause

The default Rx queue depth of UCX is 4096, but the maximum depth supported by SP670 is 4095. You need to adjust the default depth of UCX.

Solution

To solve this issue, you need to modify a line of code based on the following code:

cd /root/rpmbuild/SOURCES/
tar -zxvf ucx-1.14.1.tar.gz
vim ucx-1.14.1/src/uct/ib/base/ib_iface.c

Change the default value of RX_QUEUE_LEN from 4096 to 4095.

Package the file.

rm -rf ucx-1.14.1.tar.gz
tar zcvf ucx-1.14.1.tar.gz ucx-1.14.1

Compile and build RPM packages. In the rpmbuild directory, compile and build the ucx.spec file to generate RPM packages.
```
cd /root/rpmbuild/SPECS
rpmbuild -bb ucx.spec
```
After the build is complete, eight RPM packages are generated in the /root/rpmbuild/RPMS/aarch64 directory, as shown in the following figure.

Install the RPM packages.

cd /root/rpmbuild/RPMS/aarch64

rpm -ivh ucx-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-cma-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-debuginfo-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-debugsource-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-devel-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-ib-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-rdmacm-1.14.1-1.aarch64.rpm --force
rpm -ivh ucx-static-1.14.1-1.aarch64.rpm --force

Security Management

Routine Check Using Antivirus Software

Periodically scan clusters for viruses. This protects clusters from viruses, malicious code, spyware, and malicious programs, reducing risks such as system breakdown and information leakage. Mainstream antivirus software can be used for antivirus check.

Vulnerability Fixing

To ensure the production environment security and reduce attack risks, periodically fix the following vulnerabilities if any:

OS vulnerabilities
OpenSSL vulnerabilities
Vulnerabilities in other components

Acronyms and Abbreviations

Acronym/Abbreviation	Full Name
A - E
ECN	Explicit Congestion Notification
F - J
HPC	High Performance Compact
K - O
NUMA	Non-uniform memory access
OSD	Object Storage Daemon
P - T
PFC	Priority-based flow control
RDMA	Remote direct memory access
RoCE	RDMA over Converged Ethernet
SPDK	Storage Performance Development Kit
TCP	Transmission Control Protocol
U - Z
UCX	Unified Communication X

Change History

Date	Description
2024-09-30	This is the first official release.