Network Acceleration with the DPDK and OVS for cVMs

Enable the Data Plane Development Kit (DPDK) and Open vSwitch (OVS) for cVMs to acceleration network access. This network acceleration function is developed and commissioned on openEuler 24.03 LTS SP2. It supports LOM NIC 1 (SF221Q) and 1822 NIC SP580.

Restrictions

openEuler 24.03 has not been adapted to the MLNX_OFED driver and the DPDK cannot take over Mellanox series NICs such as CX4, CX5, and CX6.

Compile and install DPDK 23.11.

Install the dependency packages.

yum install python3-pyelftools kernel-devel kernel-headers groff autoconf automake libtool -y

Download and decompress the DPDK software package.

cd /home
wget http://fast.dpdk.org/rel/dpdk-23.11.1.tar.xz
tar -xvf dpdk-23.11.1.tar.xz

Use Meson to initialize the build configuration.

cd /home/dpdk-stable-23.11.1
meson --prefix=/usr --libdir=/usr/lib64 --bindir=/usr/bin --sbindir=/usr/sbin --includedir=/usr/include/dpdk -Dc_args="-O0 -g -fstack-protector-strong" -Dc_link_args="-g -Wl,-z,relro,-z,now,-z,noexecstack"  -Denable_kmods=true -Dplatform=generic build

Build and install the DPDK using Ninja.

ninja -C build && ninja -C build install && ldconfig

Check the installed DPDK version.
```
pkg-config --modversion libdpdk
```

Compile and install OVS 3.2.1.

Download and decompress the OVS software package.

cd /home
wget https://www.openvswitch.org/releases/openvswitch-3.2.1.tar.gz
tar -zxvf openvswitch-3.2.1.tar.gz

Generate the compilation configuration.

cd /home/openvswitch-3.2.1
./boot.sh
./configure CFLAGS="-march=armv8-a+lse+crc -fstack-protector-all -fstack-protector-strong " --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/lib64 --enable-ssl --enable-shared --with-dpdk=shared

Perform the compilation.
```
make -j`nproc` 
```
Perform the installation.
```
make -j`nproc` install
```
Check the installed OVS version.
```
ovs-vswitchd --version
```

Configure OVS to be started as a service.

cd /home/openvswitch-3.2.1
ovsdb-tool create /etc/openvswitch/conf.db
cd rhel/
cp etc_init.d_openvswitch /etc/init.d/openvswitch
chmod 755 /etc/init.d/openvswitch
mkdir -p /var/log/openvswitch/
mkdir /var/run/openvswitch/
service openvswitch restart

The OVS service has been started.

Enable memory hugepages for the host.
1. Modify the /boot/efi/EFI/openEuler/grub.cfg file.
  1. Open /boot/efi/EFI/openEuler/grub.cfg.
```
vim /boot/efi/EFI/openEuler/grub.cfg
```
  2. Press i to enter the insert mode, and add the kernel startup parameter setting default_hugepagesz=1024M hugepagesz=1024M hugepages=64 iommu.passthrough=1 to the current kernel cmdline option of the startup menu.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
2. Restart the server to access the BIOS setup screen. For details, see "Accessing the BIOS" in TaiShan Server BIOS Parameter Reference (Kunpeng 920 Processor).
  1. In the BIOS, choose Advanced > MISC Config and press Enter.
  2. Set Support Smmu to Enabled and press F10 to save the settings and exit.
  1. Check the memory hugepage setting. The hugepage granularity is 1 GB. The command output indicates that the current number of hugepages is 64. You can set an appropriate number of hugepages based on the memory size.
```
cat /proc/meminfo |grep -i huge
```

Configure the DPDK+OVS network.

Load the NIC drive module.
```
modprobe uio_pci_generic
```
Check for idle NICs and their BDF numbers.
```
ip a
```
Query NIC parameters.
```
ethtool -i enp151s0f0v0
```
Configure the DPDK to take over NICs. (Mellanox 5 does not need to be taken over.)
```
dpdk-devbind.py -b uio_pci_generic --force 0000:97:00.2
dpdk-devbind.py -s
```
The NICs have been taken over successfully.
Check the number of NUMA nodes on the host.
```
numactl -H
```
The command output shows that the host has four NUMA nodes.

Modify the OVS setting and restart the VM.

service openvswitch start
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true other_config:dpdk-socket-mem="4096,4096,4096,4096" other_config:dpdk-lcore-mask="0x1F" other_config:pmd-cpu-mask="0x1E" other_config:dpdk-pmd-driver=/lib64/librte_pmd_hinic.so other_config:vhost-iommu-support=true
service openvswitch restart

The count of 4096 in other_config:dpdk-socket-mem is equal to the number of NUMA nodes on the host.

Add the OVS network configuration script.

cd /home
vi ife_topology.sh

Add the following content to the new script and replace the example BDF number with the actual one.

#!/bin/bash
set -ex
default_br_name="br-dpdk"
default_tap_num=2

# Set the network.
set_normal() {
    if [ -z "$1" ]; then
        echo "Error: The BDF number must be specified."
        echo "Usage example: $0 normal <BDF number> [Bridge name] [Tap quantity]"
        exit 1
     fi
     local bdf=$1

    local bdf=${1:-$default_bdf}          # The first parameter: BDF number
    local br_name=${2:-$default_br_name}  # The second parameter: bridge name
    local tap_num=${3:-$default_tap_num}  # The third parameter: tap quantity

    ovs-vsctl add-br br-int -- set bridge br-int datapath_type=netdev
    ovs-vsctl add-port br-int p-tap-dpdk -- set Interface p-tap-dpdk type=patch options:peer=p-tap-dpdk-int
    ovs-vsctl add-br "$br_name" -- set bridge "$br_name" datapath_type=netdev
    ovs-vsctl add-port "$br_name" p0 -- set Interface p0 type=dpdk options:dpdk-devargs="$bdf" options:n_rxq=10
    ovs-vsctl add-port "$br_name" p-tap-dpdk-int -- set Interface p-tap-dpdk-int type=patch options:peer=p-tap-dpdk

    for i in $(seq 1 $tap_num); do
        ovs-vsctl add-br br-ply$i -- set bridge br-ply$i datapath_type=netdev
        ovs-vsctl add-port br-ply$i tap$i -- set Interface tap$i type=dpdkvhostuserclient options:vhost-server-path="/tmp/tap$i"
        ovs-vsctl add-port br-ply$i p-tap$i-int -- set Interface p-tap$i-int type=patch options:peer=p-tap$i
        ovs-vsctl add-port br-int p-tap$i -- set Interface p-tap$i type=patch options:peer=p-tap$i-int
    done
}

# Clear the network.
clean(){
    for i in `ovs-vsctl list-br`
    do
        ovs-vsctl del-br $i
    done
}
case $1 in
    clean)
        clean
        ;;
    normal)
        # Parameter sequence: BDF number → Bridge name → Tap quantity
        set_normal "$2" "$3" "$4"
        ;;
    *)
        echo "Usage: $0 {clean|normal} [bdf] [bridge_name] [tap_num]"
        echo "Example:"
        echo "  $0 normal 0000:97:00.0             # User-defined BDF number. Retain the default values for other parameters.
        echo "  $0 normal 0000:97:00.0 my-br       # User-defined BDF number and bridge name"
        echo "  $0 normal 0000:97:00.0 my-br 2     # All parameters defined by the user"
        exit 1
        ;;
esac

You can run the script in either of the following ways:

sh ife_topology.sh normal ${BDF number} ${br-name} ${tap-num}  # Set the network.
sh ife_topology.sh clean    # Clear the network.

When the normal parameter is used, the BDF number is mandatory, and the br-name and tap-num parameters are optional. The default value of br-name is br-dpdk and the default value of tap-num is 2.
When you run the sh ife_topology.sh clean command to clear the network and reconfigure the network, you need to run the service openvswitch restart command to restart the OVS service.
If an error is reported when running sh ife_topology.sh normal, stating "ovs-vsctl: Error detected while setting up 'p0': Error attaching device 'xxxx:xx:xx.x' to DPDK. See ovs-vswitchd log for details.", handle the error as follows:
1. Check that the system logs contain "dpdk|ERR|EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing."
```
journalctl -xe
```
2. Set the OVS parameter.
```
ovs-vsctl --no-wait set Open_vSwitch . other_config:iova-mode="PA"
```
3. Clear the network, restart the OVS service, and reset the network.

# Set the OVS network.
```
sh ife_topology.sh normal ${BDF number}
```

Enable network acceleration with the DPDK and OVS for the cVM.
1. Based on the VM configuration ready in 2, set the mode of access between the CPUs and memory to shared.
```
<cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='1'/>
    <numa>
        <cell id='0' cpus='0-1' memory='4' unit='GiB' memAccess='shared'/>
    </numa>
</cpu>
```
2. Specify the NIC ports to be taken over by the DPDK.
```
<interface type='vhostuser'>
    <driver name='vhost' queues='4' iommu='on'/>
    <source type='unix' path='/tmp/tap1' mode='server'/>
    <target dev='tap1'/>
    <model type='virtio'/>
</interface>
```
3. After the cVM is started, the new available NIC is displayed.
```
ip a
```
4. After the network configuration is added for the NIC, the VM can communicate with another server whose IP address is in the same network segment, and the NIC becomes available.

Parent topic: Feature Usage