Introduction

If the computation result or data of GPU 0 needs to be transmitted to GPU 1 in the same PCIe node, the communication between the two GPUs depends on the CPU and PCIe bus. That is, GPU 0 first transmits the data to the CPU, and then the CPU transmits the data to GPU 1. In this case, the data transmission bandwidth is limited by the CPU bandwidth, and the data latency is large as the data is copied twice.

To solve the preceding problems, NVIDIA proposes the GPUDirect peer to peer (P2P) technology, which is mainly used for high-speed communication between different GPUs on a single server. With this technology, a GPU can directly access the memory of the target GPU through PCIe, avoiding the need to copy data to the CPU host memory as a transit and greatly reducing the data exchange latency.

GPU P2P is both software- and hardware-dependent. That is, the hardware (such as the GPU, CPU, and mainboard) must support the P2P feature; and the software (including the BIOS, CUDA, and GPU driver) must support the P2P function, and the application must have the API to call P2P, to fully utilize the GPU bandwidth and reduce the data transmission latency between GPUs.

By default, P2P is disabled on the Kunpeng platform.

The following figure shows the single-node P2P performance evaluation.

Table 1 lists the maximum bandwidth of different PCIe x8 and x16 versions.

The Kunpeng 920 processor does not support the NVLink technology. It uses the PCIe 4.0 with a maximum bandwidth of 31.51 GB/s. By default, the GPU P2P feature is disabled. You can forcibly enable the GPU P2P feature using the following method:

1. Edit the /etc/modprobe.d/nvidia.conf file.

   1	vim /etc/modprobe.d/nvidia.conf

2. Add the following content as a new line to the end of the file:

   1	options nvidia NVreg_RegistryDwords="ForceP2P=0x11"

3. Uninstall and reinstall the NVIDIA driver.

   1 2	nvidia-uninstall ./NVIDIA-Linux-aarch64-xxx.xx.xx.run