Configuring the NIC

Purpose

If an NIC receives large amounts of requests, the NIC triggers the interrupt program to notify the kernel of new data packets. Then, the kernel invokes the interrupt handler to copy the data packets from the NIC to the memory. When the NIC has only one queue, the copying of data packets at the same time can be processed by only one core. Therefore, the NIC multi-queue mechanism is introduced. In this way, different cores can obtain data packets from different NIC queues at the same time.

When the NIC multi-queue function is enabled, the OS uses the irqbalance service to determine the CPU core that processes the network data packets in the NIC queue. If the CPU core that processes the interrupt and the NIC are not in the same NUMA node, cross-NUMA memory access is triggered. Therefore, the CPU core that processes the NIC interrupt can be set on the NUMA where the NIC is located, thereby reducing extra overheads caused by cross-NUMA memory access and improving network processing performance.

To optimize network performance, refer to the procedure below.

Procedure

1. Stop the irqbalance service on the compute node.

   1	service irqbalance stop

2. Query the number of NIC queues (assuming that the OpenStack service uses the NIC enp3s0):

   1	ethtool -l enp3s0

   

   The preceding information shows that there are 16 queues.

3. Check whether the CPU usage of the softirq process is too high. If yes, increase the number of queues. If this problem does not occur, go to the Query the NIC interrupt numbers step.
4. Dynamically adjust the number of queues to 32.

   1	ethtool -L enp3s0 combined 32

   

   More queues do not necessarily mean better performance. You need to observe the CPU usage for processing software interrupts to determine whether a performance bottleneck exists.

5. Check whether the number of software interrupts decreases.
6. Query the NIC interrupt numbers.

   1	cat /proc/interrupts | grep enp3s0

   125:         23          0      ...         enp3s0_qp0
   126:          0           0      ...         enp3s0_qp1
   ...

7. Check the NUMA node where the NIC is located:

   1	cat /sys/class/net/enp3s0/device/numa_node

   

8. Run the following command to bind each interrupt to a different CPU of the NUMA node based on the interrupt numbers. xxx indicates the interrupt number obtained in the Query the NIC interrupt numbers step.

   1	echo 0 > /proc/irq/xxx/smp_affinity_list

   

   If multiple NICs are used in the environment, it is advised to perform core binding using the preceding method for all NICs to improve the network processing performance.