Hardware Optimization Methods

NVIDIA GPUs feature strong computing capabilities but have high power consumption and generate a lot of heat. This poses high requirements on servers' power and heat dissipation.

Using More Powerful Hardware

NVIDIA A100 has different power consumption limits and memory sizes depending on the model to meet various computing requirements. If the power limit of a 250 W device is reached prematurely during computation, switch to a 300 W model for higher performance.

Using PCIe x16 Slots

A riser card installed in a PCIe x16 slot can provide higher PCIe bandwidth than when installed in a PCIe x8 slot.

Connecting Power Cables

NVIDIA GPUs are connected to riser cards through PCIe slots. In addition, power cables must be connected to ensure power supply.

Setting Server Power

GPU computing requires high power. Therefore, you are advised to use two 900 W or higher PSUs to ensure power supply and adopt the load balancing mode. In addition, do not set power capping for computing to avoid affecting the performance.

Adjusting Fan Module Speeds

You can select fan modules with stronger wind force to ensure heat dissipation and set fan module speeds to the maximum.

Log in to the iBMC using SSH.
```
ssh Administrator@IP
```
Set the fan module speed mode.
```
ipmcset -d fanmode -v 1 0
```

Set the speeds of all four fan modules to 100%.

ipmcset -d fanlevel -v 100 2 
ipmcset -d fanlevel -v 100 1 
ipmcset -d fanlevel -v 100 4 
ipmcset -d fanlevel -v 100 3

Adjusting the GPU Running Mode

nvidia-smi is a cross-platform GPU monitoring tool. It supports all Linux distributions supported by standard NVIDIA drivers and provides the functions of monitoring GPU usage and changing GPU status. This tool is provided with the graphics card driver. You can use it after the driver is installed.

Query the current information about all GPUs.

nvidia-smi

Table 1 describes the common functions of nvidia-smi.

**Table 1** Common functions of nvidia-smi
Metric	Description	Configuration Method
Fan	Fan module speed. The value of this parameter ranges from 0 to 100%, which is the theoretical fan module speed. If a fan module is blocked, its actual speed may be lower than the displayed speed. Some devices do not return fan module speeds because they do not rely on fan module cooling and keep the temperature low using other peripherals.	Adjust fan module speeds as instructed in Adjusting Fan Module Speeds.
Temp	Temperature (unit: °C).	Cannot be adjusted.
Perf	Performance status. The value ranges from P0 to P12. P0 indicates the maximum performance while P12 indicates the minimum performance.	Cannot be adjusted.
Pwr	Power consumption, including the real-time and maximum GPU power consumption.	Cannot be adjusted.
Persistence-M	Status of the persistent mode. The persistent mode consumes more power, but takes less time to start a new GPU application. The value on indicates that the persistent mode is enabled. You are advised to enable the GPU persistent mode. The GPU persistent mode is disabled by default. In this case, the GPU sleeps under a low load, and there is a possibility that the GPU cannot be woken up.	Run the following command to enable the persistent mode: nvidia-smi -pm 1
Bus-Id	GPU bus ID, which is displayed in the format of Domain:Bus:Device.Function.	Cannot be adjusted.
Disp.A	Display Active, which indicates whether the GPU is connected to the monitor (that is, whether the graphics card drives display output).	Cannot be adjusted.
Memory Usage	GPU memory usage.	Cannot be adjusted.
GPU Util	Floating GPU usage, which corresponds to the SM usage.	Cannot be adjusted.
ECC	Indicates whether the error correcting code (ECC) function is enabled.	Enables or disables ECC. nvidia-smi -e 0/1 The value 0 means disabling the function, and the value 1 indicates enabling the function. Reset the ECC error count. nvidia-smi -p 0/1 The value 0 indicates VOLATILE, and the value 1 indicates AGGREGATE.
Compute M	Compute mode.	Switch the compute mode. nvidia-smi -c 0/1/2 The value 0 indicates Default, 1 indicates Exclusive_Process, and 2 indicates Prohibited.

The lower part of the preceding figure displays the GPU memory usage of each process. Some metrics cannot be obtained from the preceding command output, but have a great impact on the performance.

Set the GPU working frequency. The following command sets the GPU memory clock to 1215 MHz and graphics clock to 1410 MHz.
```
nvidia-smi -ac 1215,1410
```
Set the maximum graphics clock of the first two GPUs to 1410 MHz.
```
nvidia-smi -lgc 1410,1410
```
Ensure that the firmware versions of all GPUs are the same. Otherwise, the performance may be affected.
```
nvidia-smi -q |grep -i VBIOS
```

To view GPU details in real time, run the nvidia-smi dmon command.

GPU statistics are displayed in a row scrolling format. The display of metrics to be monitored can be adjusted based on the width of the terminal window. A maximum of four GPUs can be monitored. If no GPU is specified, GPUs 0 to 3 are monitored by default (the GPU index starts from 0).

Table 2 describes the common additional options.

**Table 2** Additional options of nvidia-smi dmon
Command	Function
nvidia-smi dmon -i xxx	Specifies the GPU index, PCI bus ID, or PCI bus UUID. Use commas (,) to separate them.
nvidia-smi dmon -d xxx	Specifies the refresh interval (1 second by default).
nvidia-smi dmon -c xxx	Displays the specified number of statistical records and exits.
nvidia-smi dmon -s xxx	Specifies the monitoring metrics to be displayed. The default value is puc. The options are as follows: p: power usage and temperature (pwr: power consumption; temp: temperature). u: GPU usage (sm: SM; mem: memory; enc: encoding resources; dec: decoding resources). c: GPU processor and memory clocks (mclk: memory clock frequency; pclk: processor clock). v: power supply and thermal exceptions. m: frame buffer (FB) memory and BAR1 memory. e: number of ECC errors and PCIe replay errors. t: PCIe read/write bandwidth.
nvidia-smi dmon -o D/T	Specifies the time format (D: YYYYMMDD; T: HH:MM:SS).
nvidia-smi dmon -f xxx	Outputs the queried information to a specific file, and does not display the information on the terminal.

In many cases, the device monitoring information is not useful. In this case, you can run the nvidia-smi pmon command to monitor the GPU process. The command output displays the GPU process status information in a row scrolling format. Table 3 describes common additional options of the command.

**Table 3** Additional options of nvidia-smi pmon
Command	Function
nvidia-smi pmon -i xxx	Specifies the GPU index, PCI bus ID, or PCI bus UUID. Use commas (,) to separate them.
nvidia-smi pmon -d xxx	Specifies the refresh interval (1 second by default).
nvidia-smi pmon -c xxx	Displays the specified number of statistical records and exits.
nvidia-smi pmon -s xxx	Specifies the monitoring metrics to be displayed. The default value is u. The options are as follows: u: GPU usage. m: FB memory usage.
nvidia-smi pmon -o D/T	Specifies the time format (D: YYYYMMDD; T: HH:MM:SS).
nvidia-smi pmon -f xxx	Outputs the queried information to a specific file, and does not display the information on the terminal.

Configuring Air Channels and Improving Heat Dissipation

Heat dissipation has a great impact on GPU performance. During computing, you can remove the drives from the front panel to allow greater air intake for fan modules. If multiple GPUs are used, evenly configuring GPUs in the slots will bring certain performance benefits.

Parent topic: GPU Optimization Methods