Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Bonding

  1. Create VFs on the host.
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
    		 
    echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
ip link set enp1s0f0 vf 0 mac e4:11:22:33:61:11
ip link set enp1s0f0 vf 1 mac e4:11:22:33:61:22
ip link set enp1s0f0 vf 2 mac e4:11:22:33:61:33
ip link set enp1s0f0 vf 3 mac e4:11:22:33:61:44
echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind
devlink dev eswitch set pci/0000:01:00.0 mode switchdev
echo 4 > /sys/class/net/enp1s0f1/device/sriov_numvfs
ip link set enp1s0f1 vf 0 mac e4:11:22:33:62:11
ip link set enp1s0f1 vf 1 mac e4:11:22:33:62:22
ip link set enp1s0f1 vf 2 mac e4:11:22:33:62:33
ip link set enp1s0f1 vf 3 mac e4:11:22:33:62:44
echo 0000:01:01.2 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:01.3 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:01.4 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:01:01.5 > /sys/bus/pci/drivers/mlx5_core/unbind
devlink dev eswitch set pci/0000:01:00.1 mode switchdev
    • For a Linux bond, after setting pci mode to switchdev, strictly follow the operation procedure. Do not rebind the VFs immediately. Otherwise, an error is reported.
    • Though it is allowed to create a bond immediately after the system is started, but it is not advised to do so. You can set onboot to no in the ifcfg-xxxxx file.
    • Each MAC address must be unique on the local server, the peer server, and the switch.
  2. Create a Linux bond.
    1. Modify the configuration files of physical network ports enp1s0f0 and enp1s0f1. 
      1
2
      		 
      vim /etc/sysconfig/network-scripts/ifcfg-enp1s0f0
vim /etc/sysconfig/network-scripts/ifcfg-enp1s0f1
    2. Press I to enter the insert mode and add the following content: 
      1
2
      		 
      MASTER=bond0
SLAVE=yes

    3. Press Esc to exit the insert mode. Input :wq! and press Enter to save the file and exit.
    4. Add a configuration file for the bond port. 
      1
      		 
      vim /etc/sysconfig/network-scripts/ifcfg-bond0
    5. Press I to enter the insert mode and add the following content: 
      1
2
3
4
5
6
7
8
      		 
      DEVICE=bond0
NAME='bond0'
TYPE=Ethernet
NM_CONTROLLED=no
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS='mode=4 miimon=100 xmit_hash_policy=layer3+4'
IPV6INIT=no
      • In the BONDING_OPTS configuration:

        mode indicates the mode. The value 4 indicates LACP.

        miimon=100 indicates that the monitoring is performed every 100 ms.

        xmit_hash_policy=layer3+4 indicates the LACP balancing policy, which uses Layer 3 and Layer 4 balancing for IP addresses and ports.

      • The active/standby mode parameter is mode=1 miimon=100.
    6. Press Esc to exit the insert mode. Input :wq! and press Enter to save the file and exit.
    7. Load the bond kernel module. 
      1
      		 
      modprobe bonding mode=4 miimon=100

      If needed, set mode to 1 for the active/standby mode, and to 4 for the LACP mode.

    8. Start the bond. 
      1
      		 
      ifup bond0

      Do not restart the network.

    9. Check the bond configuration. 
      cat /proc/net/bonding/bond0

  3. Bind the VFs.
    1
2
3
4
5
6
7
8
    		 
    echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:01.2 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:01.3 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:01.4 > /sys/bus/pci/drivers/mlx5_core/bind
echo 0000:01:01.5 > /sys/bus/pci/drivers/mlx5_core/bind
  4. Start the OVS and configure the network.
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
    		 
    systemctl start openvswitch
ovs-vsctl add-br br-ovs
ovs-vsctl add-port br-ovs bond0
ovs-vsctl add-port br-ovs enp1s0f0_0
ovs-vsctl add-port br-ovs enp1s0f0_1
ovs-vsctl add-port br-ovs enp1s0f0_2
ovs-vsctl add-port br-ovs enp1s0f0_3
ovs-vsctl add-port br-ovs enp1s0f1_3
ovs-vsctl add-port br-ovs enp1s0f1_2
ovs-vsctl add-port br-ovs enp1s0f1_1
ovs-vsctl add-port br-ovs enp1s0f1_0
ifconfig bond0 up
ip link set dev enp1s0f0_0 up
ip link set dev enp1s0f0_1 up
ip link set dev enp1s0f0_2 up
ip link set dev enp1s0f0_3 up
ip link set dev enp1s0f1_0 up
ip link set dev enp1s0f1_1 up
ip link set dev enp1s0f1_2 up
ip link set dev enp1s0f1_3 up
  5. Start a VM and log in to it.
    1
2
    		 
    virsh start vm1
virsh console vm1
  6. Send packets between VMs.
    • Run the following command on VM 1 of host 2: 
      1
      		 
      iperf3 -s
    • Run the following command on VM 1 of host 1: 
      1
      		 
      iperf3 -c <Host2vm1_ip> -t 0
  7. Press Ctrl+] to exit from the VM. View the flow table offloading on any of the physical machines.
    1
    		 
    watch -n 1 -d ovs-appctl dpctl/dump-flows type=offloaded

    Verify the bidirectional flow table offloading. In LACP mode, verify that the total bandwidth of the eight VMs is close to the total bandwidth of the two network ports. In active/standby mode, verify the network connectivity when one network port is disconnected.

  8. Delete the bond.
    1. Stop the VM. 
      1
      		 
      virsh shutdown vm1
    2. Delete the OVS bridge. 
      1
      		 
      ovs-vsctl del-br br-ovs
    3. Unbind all related VFs (for details, see 1) or clear the VFs. 
      1
2
      		 
      echo 0 > /sys/class/net/enp1s0f0/device/sriov_numvfs
echo 0 > /sys/class/net/enp1s0f1/device/sriov_numvfs
    4. Delete the bond. 
      1
2
      		 
      ip link delete bond0
rmmod bonding