Configuring the High-Speed NICs

Principle

In HPC scenarios, the InfiniBand network solution is used to provide high bandwidth and low latency. In large-scale HPC application scenarios, you should ensure that the InfiniBand NIC is inserted into the correct PCIe slot to prevent the InfiniBand NIC speed from decreasing due to insufficient PCIe signals and ensure application performance.

Procedure

1. Run the following command to check whether the PCIe IDs of the InfiniBand NICs are consistent. Ensure that the InfiniBand NICs of each node in the cluster are inserted into the same slots.

   lspci |grep -i mellanox

   Figure 1 shows the command output.

   Figure 1 Result example
   
2. Run the following command to query the PCIe NIC status. Ensure that the NIC status is Speed 16GT/s, Width x16 (for PCIe 4.0) or Speed 8GT/s, Width x16 (for PCIe 3.0).

   lspci -vvv -s 01:00.0

   Figure 2 shows the command output.

   Figure 2 Result example