Uneven PG Distribution in a Ceph Cluster

Symptom

When the drives are heavy-loaded during the I/O test, the load of some drives reaches 100%, while that of some drives is less than 80%. The overall drive load is unbalanced. After the ceph pg dump command is executed, it is found that the placement groups (PGs) are not evenly distributed in the Ceph cluster.

Key Process and Cause Analysis

The number of PGs on each OSD varies greatly. Some OSDs are overloaded and become bottlenecks.

Conclusion and Solution

Use the balancer plugin to optimize PG distribution.

Check the PG distribution.
- Method 1:
```
ceph balancer eval
```
- Method 2:
```
ceph pg dump
```
Use either of the preceding commands.
Enable automatic balancing for Ceph PGs.
```
ceph balancer mode upmap
ceph balancer on
```
Ceph adjusts PG distribution every 60 seconds.
Repeat step 1. If the PG distribution does not change, the PG distribution is optimal.