Binding NIC Interrupts to Cores

Purpose

Compared with using irqbalance of the kernel to schedule NIC interrupts on all cores, using manual core binding to fix interrupts can effectively improve the capability of the service network to receive and send packets.

Procedure

Disable irqbalance.
Before binding cores to NICs, disable irqbalance.
- Stop the irqbalance service. The setting will be invalid after the system restarts.
  1
  systemctl stop irqbalance.service
- Disable the irqbalance service. This setting is permanently valid.
  1
  systemctl disable irqbalance.service
- Check whether the irqbalance service is disabled.
  1
  systemctl status irqbalance.service

Create a script for binding interrupts to cores.

Create a script file named bind_irq.sh.
```
vi bind_irq.sh
```

Press i to enter the insert mode and add the following content to the file:

#!/bin/bash
# set -x

net_name=$1
irq_cores=$2
irq_cnt=$3
cpu_cores=""
numa_nodes=""
bus_info=""
core_num_per_node=""
net_numa=""
irq_id_array=""
irq_core_list=""
irq_core_list_sz=""

function get_info() {
    if [[ ${net_name} == "" ]]; then
        echo "please specify the network interface parameter"
        exit
    fi

    max_irq_cnt=$(ethtool -l ${net_name} | grep "^Combined:" | head -n1 | awk '{print $2}')
    if [[ ${irq_cnt} == "" ]]; then
        irq_cnt=$(ethtool -l ${net_name} | grep "^Combined:" | head -n1 | awk '{print $2}')
    elif [[ ${irq_cnt} -gt ${max_irq_cnt} ]]; then
        echo "the max combined is ${max_irq_cnt}, ${irq_cnt} is too large"
        exit
    fi

    if [[ ${irq_cnt} == "" ]]; then
        echo "get combined number failed"
        exit
    fi

    cpu_cores=`nproc`
    numa_nodes=$(lscpu | grep "NUMA node(s):\|NUMA node: " | awk '{print $3}')
    bus_info=$(ethtool -i ${net_name} | grep "^bus-info:" | awk '{print $2}')
    ((core_num_per_node = ${cpu_cores} / ${numa_nodes}))
    net_numa=$(lspci -vvvs ${bus_info} | grep "NUMA node:" | awk '{print $3}')

    echo "cpu cores: ${cpu_cores}"
    echo "cpu numa nodes: ${numa_nodes}"
    echo "cpu cores per numa: ${core_num_per_node}"
    echo "network interface: ${net_name}"
    echo "combined: ${irq_cnt}"
    echo "businfo: ${bus_info}"
    echo "network numa: ${net_numa}"
}

function parse_cpu_list() {
    IFS_bak=$IFS
    IFS=','
    cpurange=($1)
    IFS=${IFS_bak}
    irq_core_list=()
    n=0
    for i in ${cpurange[@]};do
        start=`echo $i | awk -F'-' '{print $1}'`
        stop=`echo $i | awk -F'-' '{print $NF}'`
        for x in `seq $start $stop`;do
            irq_core_list[$n]=$x
            let n++
        done
    done
}

function get_irq_cores_list() {
    if [[ ${irq_cores} != "" ]]; then
        parse_cpu_list ${irq_cores}
    elif [[ ${irq_cnt} -ge ${core_num_per_node} ]]; then
        ((cpu_start=${core_num_per_node}*${net_numa}))
        ((cpu_end=$cpu_start+${core_num_per_node}-1))
        irq_cores="${cpu_start}-${cpu_end}"
        parse_cpu_list ${irq_cores}
    else
        ((cpu_end=${core_num_per_node}*${net_numa}+${core_num_per_node}-1))
        ((cpu_start=${cpu_end}-${irq_cnt}+1))
        irq_cores="${cpu_start}-${cpu_end}"
        parse_cpu_list ${irq_cores}
    fi

    echo "irq cpu cores:" $(printf "%s" "${irq_core_list[*]}")
    irq_core_list_sz=${#irq_core_list[@]}
}

function set_irq() {
    # Set the NIC queue depth to the value specified by the parameter. If not specified, the maximum value is used by default.
    ethtool -L ${net_name} combined ${irq_cnt}

    # Obtain the interrupt ID based on the NIC.
    # irq_ids=`cat /proc/interrupts| grep -E ${bus_info} | head -n$irq_cnt | awk -F ':' '{print $1}'`
    irq_ids=`grep "${net_name}" /proc/interrupts | awk -F ':' '{print $1}'`
    irq_ids=`echo ${irq_ids}`
    echo "irq ids: ${irq_ids}"
    # Convert the irq_ids string into the irq_id_array array.
    IFS=' ' read -r -a irq_id_array <<< "${irq_ids}"
}

function bind_irq() {
    for((i=0;i<irq_cnt;i++));do
        irq=${irq_id_array[i]}
        core=${irq_core_list[$((i%irq_core_list_sz))]}
        echo "${i}: ${irq} -> ${core}"
        echo ${core} > /proc/irq/${irq}/smp_affinity_list
    done
}

get_info
get_irq_cores_list
set_irq
bind_irq

Press Esc, type :wq!, and press Enter to save the file and exit.

Run the bind_irq.sh script to bind interrupts to cores for a specified NIC.
The script can be used in the following ways:
- (Recommended) Method 1: Bind the specified NIC interrupt to the NUMA node where the NIC resides and set the number of NIC queues to the maximum value. For example, if the NIC name is enp131s0, run the following command to use the script:
  
  Run ifconfig to view the NIC name.
```
bash bind_irq.sh enp131s0
```
- Method 2: Bind the specified NIC interrupt to a CPU core range and set the number of NIC queues to the maximum value.
  For example, the interrupt of the NIC named enp131s0 is bound to the CPU cores within the 18-31, 60-63, 92-95, 124-127 range.
  
  Run lscpu to view the available CPU ID range.
```
bash bind_irq.sh enp131s0 "18-31,60-63,92-95,124-127"
```
- Method 3: Bind the specified NIC interrupt to a CPU core range and set the number of NIC queues to a specific value. The number of NIC queues must be less than or equal to the maximum number of NIC queues.
  
  Run ethtool -l enp131s0 to query the maximum number of queues of the enp131s0 NIC.
```
bash bind_irq.sh enp131s0 "18-31,60-63,92-95,124-127" 32
```

Create an irqCheck.sh script.

Create irqCheck.sh.
```
vi irqCheck.sh
```

Press i to enter the insert mode and add the following content to the file:

#!/bin/bash
# NIC name
intf=$1
log=irqSet-`date "+%Y%m%d-%H%M%S"`.log
# Number of available CPUs
cpuNum=$(cat /proc/cpuinfo |grep processor -c)
# RX and TX interrupt lists
irqListRx=$(cat /proc/interrupts | grep ${intf} | awk -F':' '{print $1}')
irqListTx=$(cat /proc/interrupts | grep ${intf} | awk -F':' '{print $1}')
# Bind the RX interrupt requests (IRQs).
for irqRX in ${irqListRx[@]}
do
cat /proc/irq/${irqRX}/smp_affinity_list
done
# Bind the TX IRQs.
for irqTX in ${irqListTx[@]}
do
cat /proc/irq/${irqTX}/smp_affinity_list
done

Press Esc, type :wq!, and press Enter to save the file and exit.

Run the following command to check whether the core binding is successful.
1
sh irqCheck.sh enp131s0
An example of the expected result:

Parent topic: Hardware Tuning