NUMA Affinity

1:1 CPU Core Binding and Same-Die Memory Access

vCPUs can be bound to CPUs in the same processor or CPUs in the same NUMA node. Avoid cross-die and cross-chip memory access of a Docker container to prevent performance deterioration. By default, vCPUs of different containers may run on the same physical CPU, which causes CPU resource contention and frequent VM ID changes. As a result, L1 TLB flushing frequently occurs and the TLB miss rate is high, causing performance deterioration.

1. Query the NUMA information.

   1	numactl -H

   

   The preceding figure shows the CPU core distribution of Kunpeng 920 processors. CPU cores 0 to 23 are in NUMA node 0, CPU cores 24 to 47 in NUMA node 1, CPU cores 48 to 71 in NUMA node 2, and CPU cores 72 to 95 in NUMA node 3. When binding Docker container vCPUs to cores, avoid cross-die and cross-chip memory access to prevent performance deterioration.

2. Bind each container vCPU to a CPU core and assign memory in the same NUMA node to each vCPU.

   The Kunpeng 920 processor is used as an example. Create a container named 4u8g_01, allocate 4 cores (bound to CPU cores 4 to 7 of NUMA 0) and 8 GB memory, use the centos:latest image, mount a local volume, and map /home on the local host to /home on the container.

   1	docker run -d -it --cpus=4 --cpuset-cpus=4-7 --cpuset-mems=0 -m 8192m --name 4u8g_01 -v /home:/home centos:latest

   

   For details about the docker run command, see docker container run.

   Command format: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

   Parameters in the command are described as follows:

   • -d: runs the container in the background and prints the container ID.
   • -i: enables the STDIN even if no attach operation is performed.
   • -t: allocates a pseudo-TTY.
   • --cpus: specifies the number of CPU cores to be allocated.
   • --cpuset-cpus: specifies the CPU cores to be bound.
   • --cpuset-mems: specifies the NUMA node.
   • -m: specifies the memory size to be allocated.
   • --name: specifies the Docker container name.
   • -v: specifies the volume to be bound.
   • centos:latest: specifies the local image. centos indicates the repository, and latest indicates the tag.

Cross–CPU Cluster Binding and Same-Die Memory Access

Each Kunpeng 920 series processor provides two super CPU clusters (SCCLs). Each SCCL contains six to eight CPU clusters, and each CPU cluster contains four CPU cores. When binding CPU cores to a Docker container, you are advised to use CPU cores across multiple CPU clusters to improve the Docker container performance. This method can reduce bandwidth bottlenecks of the L3 cache caused by contention among cores in the same CPU cluster.

Binding vCPUs of a Docker container to CPU cores across multiple CPU clusters can bring the following benefits:

• When the load is light, the memory bandwidth can be fully utilized.
• Contention for L3 cache tags can be dramatically reduced and the memory bandwidth and CPU computing performance can be improved.

If the number of clusters is greater than the number of vCPUs (N) of the container to be created, you can select any N clusters.

Set cross–CPU cluster binding and same-NUMA memory access.

The Kunpeng 920 processor is used as an example. Create a container named 8u16g_02, allocate 8 cores (bound to CPU cores 3, 4, 8, 9, 12, 16, 20, and 21 of NUMA 0) and 16 GB memory, use the centos:latest image, mount a local volume, and map /home on the local host to /home on the container.

1

docker run -d -it --cpus=8 --cpuset-cpus=3,4,8,9,12,16,20,21 --cpuset-mems=0 -m 16384m --name 8u16g_02 -v /home:/home centos:latest

For details about the docker run command, see docker container run.

Command format: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

The parameters are described as follows:

• -d: runs the container in the background and prints the container ID.
• -i: enables the STDIN even if no attach operation is performed.
• -t: allocates a pseudo-TTY.
• --cpus: specifies the number of CPU cores to be allocated.
• --cpuset-cpus: specifies the CPU cores to be bound.
• --cpuset-mems: specifies the NUMA node.
• -m: specifies the memory size to be allocated.
• --name: specifies the Docker container name.
• -v: specifies the volume to be bound.
• centos:latest: specifies the local image. centos indicates the repository, and latest indicates the tag.