Ceph Tuning
- Purpose
Adjust the Ceph configuration items to fully utilize the hardware performance of the system.
- Procedure
You can edit the /etc/ceph/ceph.conf file to modify all Ceph configuration parameters.
For example, to change the number of copies to 4, you can add osd_pool_default_size = 4 to the /etc/ceph/ceph.conf file and run the systemctl restart ceph.target command to restart the Ceph daemon process for the change to take effect.
The setting takes effect only for the current Ceph node. To enable the settings of the entire Ceph cluster to take effect, you need to modify the ceph.conf file of each Ceph node and restart the Ceph daemon process.
Table 1 lists the optimization items.
Table 1 Ceph parameters Parameter
Description
Suggestion
[global]
osd_pool_default_min_size
Specifies the minimum number of I/O copies that the PG can receive. If a PG is in the degraded state, its I/O capability is not affected.
Default value: 0
Recommended value: 1
cluster_network
Configures a network segment different from the public network. This network segment is used for replication and data balancing between OSDs to relieve the pressure on the public network.
Default value: none
Recommended value: 192.168.4.0/24
osd_pool_default_size
Specifies the number of copies.
Default value: 3
Recommended value: 3
mon_max_pg_per_osd
Indicates the PG alarm threshold. You can increase the value for better performance.
Default value: 250
Recommended value: 3000
[rgw]
rgw_override_bucket_index_max_shards
Specifies the number of shards per bucket index. The value 0 indicates that no shard is available.
Default value: 0
Recommended value: 8
PG Distribution Tuning
- Purpose
Adjust the number of PGs on each OSD to balance the load on each OSD.
- Procedure
By default, Ceph allocates eight PGs/PGPs to each storage pool. When creating a storage pool, run the ceph osd pool create {pool-name} {pg-num} {pgp-num} command to specify the number of PGs/PGPs, or run the ceph osd pool set {pool_name} pg_num {pg-num} and ceph osd pool set {pool_name} pgp_num {pgp-num} commands to change the number of PGs/PGPs created in a storage pool. After the modification, run the ceph osd pool get {pool_name} pg_num/pgp_num command to check the number of PGs/PGPs in the storage pool.
The default value of ceph balancer mode is none. You can run the ceph balancer mode upmap command to change it to upmap. The Ceph balancer function is disabled by default. You can run the ceph balancer on/off command to enable or disable the Ceph balancer function.
Table 2 describes the PG distribution parameters.
Table 2 PG distribution parameters Parameter
Description
Suggestion
pg_num
Total PGs = (Total_number_of_OSD x 100)/max_replication_count
Round up the result to the nearest integer power of 2.
Default value: 8
Symptom: A warning is displayed if PGs are insufficient.
Suggestion: Calculate the value based on the formula.
pgp_num
Sets the number of PGPs to be the same as that of PGs.
Default value: 8
Symptom: It is recommended that the number of PGPs be the same as that of PGs.
Suggestion: Calculate the value based on the formula.
ceph_balancer_mode
Enables the balancer plug-in and sets the plug-in mode to upmap.
Default value: none
Symptom: If PGs are not evenly distributed across OSDs, some OSDs may be overloaded and become bottlenecks.
Suggestion: Set this parameter to upmap.
- The number of PGs carried by each OSD must be the same or close. Otherwise, some OSDs may be overloaded and become bottlenecks. The balancer plug-in can be used to optimize the PG distribution. You can run the ceph balancer eval or ceph pg dump command to view the PG distribution.
- Run the eph balancer mode upmap and ceph balancer on commands to automatically optimize Ceph PG distribution. Ceph adjusts the distribution of a few PGs every 60 seconds. Run the ceph balancer eval or ceph pg dump command to view the PG distribution. If the PG distribution does not change, the distribution is optimal.
- The PG distribution of each OSD affects the load balancing of write data. In addition to optimizing the number of PGs carried by each OSD, the distribution of the primary PGs also needs to be optimized. That is, the primary PGs need to be distributed to each OSD as evenly as possible.