Binding a Single NUMA Node

To optimize the MySQL performance, you can bind the MySQL pod of a compute node to a single NUMA node. K8s provides two CPU allocation policies: CPU Manager and Topology Manager.

Binding a single NUMA node: Perform steps 1 to 4 on each compute node.

Removing the restriction on a single NUMA node: Perform 1 to 4 on each compute node. Retain default configuration of the configuration file in 2.

Confirm the K8s version and the actual CPU binding requirements.
Run the kubectl version command to query the K8s version.
- If the K8s version is 1.16 or later, both the CPU Manager policy and the Topology Manager policy can be used for this version.
- If the K8s version is 1.12 or later, only the CPU Manager policy can be used for this version.
  CPU binding results in the exclusive occupation of the CPU. Check whether the CPU needs to be exclusively occupied before CPU binding.

Modify the Kubelet configuration file.

Open the configuration file.

1	vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

The default content of the configuration file is as follows:

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
 
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Press i to enter the insert mode and edit the file as follows:

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet

# Modification 1: Add the two lines of the ExecStartPre configuration to the file.
ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
 
ExecStart=

# Modification 2: Add parameters such as --kube-reserved, --cpu-manager-policy, --feature-gates, and --topology-manager-policy to the end of the ExecStart configuration.
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --kube-reserved=cpu=2,memory=250Mi --cpu-manager-policy=static --feature-gates=CPUManager=true --topology-manager-policy=single-numa-node

CPU Manager policy: --feature-gates=CPUManager=true. Ensure that the number of CPU cores specified by limits of mysql-1, mysql-2, and mysql-3 in the YAML file is allocated in a continuous manner.
Topology Manager policy: --topology-manager-policy=single-numa-node. Ensure that the number of CPU cores specified by limits of mysql-1, mysql-2, and mysql-3 in the YAML file is bound to a single NUMA node. If the number of CPU cores needs to be bound to a single NUMA node, the number of CPU cores specified by limits in the YAML file must be less than or equal to the number of CPU cores of a single NUMA node. You can run the lscpu or numactl -H command to check the number of CPU cores of each NUMA node. Otherwise, after creating and deploying a MySQL pod on the K8s primary node and then running the watch kubectl get pod -n ns-mysql-test -o wide command, you will find that the pod fails to be created.
In actual scenarios, it is assumed that the number of CPU cores of the created MySQL pod is greater than the number of CPU cores of a single NUMA node and less than or equal to the number of CPU cores of 1P (in this document, 1P corresponds to two NUMA nodes), and the NUMA nodes cannot cross P nodes. In this case, delete --topology-manager-policy=single-numa-node from the YAML file and delete the resource limit parameter limits from the file on the primary node. After using K8s to create and deploy a MySQL pod on the primary node, run the taskset -pac command on the compute node to manually bind the MySQL process and threads to CPU cores 0 to 47 (NUMA node 0 and NUMA node 1). To perform CPU binding, perform the following steps:
1. Check the ID of the MySQL process.
  1
  ps -ef | grep mysql
2. Check the CPU cores to which the MySQL process is bound.
  1
  taskset -pac mysql process_ID
3. Bind the MySQL process and threads to CPU cores 0 to 47 (NUMA node 0 and NUMA node 1).
  1
  taskset -pac 0-47 mysql process_ID
4. Check the CPU cores to which the MySQL process is bound.
  1
  taskset -pac mysql process_ID

Press Esc, type :wq!, and press Enter to save the file and exit.

Delete the CPU management status file cpu_manager_state.
1
rm -f /var/lib/kubelet/cpu_manager_state

Restart the kubelet service.

systemctl daemon-reload && systemctl restart kubelet

Check the kubelet status.

1	systemctl status kubelet

Modify the mysql_deployment.yaml configuration file.

Select proper CPU and memory configuration values based on the CPUs and memory of the node to be deployed. For example, the physical machine has four NUMA nodes, 1P has two NUMA nodes, and each NUMA node has 24 CPU cores in this document. The CPU core configuration is as follows:

......
spec:
  nodeSelector:
    test: "mysql-test-1"
  containers:
  - name: mysql-1
    image: mymysql/centos8-mysql-arm:8.0.19
    resources:
      limits:
        cpu: 16
        memory: 64Gi
    ......
---
......
spec:
  nodeSelector:
    test: "mysql-test-2"
  containers:
  - name: mysql-2
    image: mymysql/centos8-mysql-arm:8.0.19
    resources:
      limits:
        cpu: 16
        memory: 64Gi
    ......
---
......
spec:
  nodeSelector:
    test: "mysql-test-3"
  containers:
  - name: mysql-3
    image: mymysql/centos8-mysql-arm:8.0.19
    resources:
      limits:
        cpu: 16
        memory: 64Gi
    ......

To enable the single-numa-node mode for a pod, you must configure the CPU and memory in resources of the pod, and ensure that the value of requests is the same as the value of limits in resources (that is, Guaranteed Pod). This document does not describe the requests configuration, that is, the value of requests is the same as that of limits by default.

Redeploy mysql_deployment.yaml on the primary node.

kubectl delete -f ./mysql_deployment.yaml
kubectl create -f ./mysql_deployment.yaml

Check the usage of pods and NUMA nodes.

docker ps -a | grep mysql

bcc93653c574        48858e629fa6           "/entrypoint.sh mysq..."   31 minutes ago      Up 31 minutes                                 k8s_mysql-2_mysql-2_ns-mysql-test_605956f2-1e13-49c6-a197-6220915130bc_0
ea9afa2c2104        k8s.gcr.io/pause:3.2   "/pause"                 32 minutes ago      Up 32 minutes                                 k8s_POD_mysql-2_ns-mysql-test_605956f2-1e13-49c6-a197-6220915130bc_0

docker inspect bcc93653c574 | grep Cpuset

            "CpusetCpus": "2-17",
            "CpusetMems": "",

lscpu

Architecture:          aarch64
Byte Order:            Little Endian
CPU(s):                96
On-line CPU(s) list:   0-95
Thread(s) per core:    1
Core(s) per socket:    48
Socket(s):             2
NUMA node(s):          4
Model:                 0
CPU max MHz:           2600.0000
CPU min MHz:           200.0000
BogoMIPS:              200.00
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              49152K
NUMA node0 CPU(s):     0-23
NUMA node1 CPU(s):     24-47
NUMA node2 CPU(s):     48-71
NUMA node3 CPU(s):     72-95
Flags:                 fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop

The preceding information indicates that mysql-1 is restricted to run on NUMA node 0.

Parent topic: K8s MGR Maintenance