Ceph Tuning
Tuning Ceph Configuration
- Purpose
Modify the Ceph configuration to maximize system resource utilization.
- Procedure
You can modify the Ceph configuration in the /etc/ceph/ceph.conf file. For example, to change the number of copies to 4, you can add osd_pool_default_size = 4 to the /etc/ceph/ceph.conf file and run the systemctl restart ceph.target command to restart the Ceph daemon process for the change to take effect.
The setting takes effect only for the current Ceph node. To enable the settings of the entire Ceph cluster to take effect, you need to modify the ceph.conf file of each Ceph node and restart the Ceph daemon process. Table 1 describes the Ceph parameters to be modified.
Table 1 Ceph parameters Parameter
Description
Suggestion
[global]
cluster_network
Configures a network segment different from the public network. This network segment is used for replication and data balancing between OSDs to relieve the pressure on the public network.
Configure a network segment that is different from the public network segment and set the value to, for example, 192.168.4.0/24.
public_network
Configure a network segment that is different from the cluster network segment and set the value to, for example, 192.168.3.0/24.
Table 2 describes other parameters that can be modified.
Table 2 Other parameters Parameter
Description
Suggestion
[global]
osd_pool_default_min_size
Specifies the minimum number of I/O copies that the PG can receive. If a PG is in the degraded state, its I/O capability is not affected.
Default value: 0
Recommended value: 1
osd_pool_default_size
Specifies the number of copies.
Default value: 3
Recommended value: 3
osd_memory_target
Specifies the size of memory that each OSD process is allowed to obtain.
Default value: 4294967296
Recommended value: 4294967296
[mon]
mon_clock_drift_allowed
Specifies the clock drift between MONs.
Default value: 0.05
Recommended value: 1
mon_osd_min_down_reporters
Specifies the minimum number of down OSDs that triggers a report to the MONs.
Default value: 2
Recommended value: 13
mon_osd_down_out_interval
Specifies the duration (in seconds) for which Ceph waits before an OSD is marked as down or out.
Default value: 600
Recommended value: 600
[OSD]
osd_journal_size
Specifies the OSD journal size.
Default value: 5120
Recommended value: 20000
osd_max_write_size
Specifies the maximum size (in MB) of data that can be written by an OSD at a time.
Default value: 90
Recommended value: 512
osd_client_message_size_cap
Specifies the maximum size (in bytes) of data that can be stored in the memory by the clients.
Default value: 100
Recommended value: 2147483648
osd_deep_scrub_stride
Specifies the number of bytes that can be read during deep scrubbing.
Default value: 524288
Recommended value: 131072
osd_map_cache_size
Specifies the size of the cache (in MB) that stores the OSD map.
Default value: 50
Recommended value: 1024
osd_recovery_op_priority
Specifies the priority of the restoration operation. The value ranges from 1 to 63. A larger value indicates higher resource usage.
Default value: 3
Recommended value: 2
osd_recovery_max_active
Specifies the maximum number of active restoration requests allowed at the same time.
Default value: 3
Recommended value: 10
osd_max_backfills
Specifies the maximum number of backfills allowed by an OSD.
Default value: 1
Recommended value: 4
osd_min_pg_log_entries
Specifies the maximum number of PGLogs that can be recorded when the PG is normal.
Default value: 3000
Recommended value: 30000
osd_max_pg_log_entries
Specifies the maximum number of PGLogs that can be recorded when the PG is degraded.
Default value: 3000
Recommended value: 100000
osd_mon_heartbeat_interval
Specifies the interval (in seconds) for an OSD to ping a MON.
Default value: 30
Recommended value: 40
ms_dispatch_throttle_bytes
Specifies the maximum number of messages to be dispatched.
Default value: 10485760
Recommended value: 1048576000
objecter_inflight_ops
Specifies the maximum number of unsent I/O requests allowed. This parameter is used for client traffic control. If the number of unsent I/O requests exceeds the threshold, the application I/O is blocked. The value 0 indicates that the number of unsent I/O requests is not limited.
Default value: 1024
Recommended value: 819200
osd_op_log_threshold
Specifies the number of operation logs displayed at a time.
Default value: 5
Recommended value: 50
osd_crush_chooseleaf_type
Specifies the bucket type when the CRUSH rule uses chooseleaf.
Default value: 1
Recommended value: 0
journal_max_write_bytes
Specifies the maximum number of bytes that can be written to a journal at a time.
Default value: 1048560
Recommended value: 1073714824
journal_max_write_entries
Specifies the maximum number of records that can be written to a journal at a time.
Default value: 100
Recommended value: 10000
[Client]
rbd_cache
Specifies the RBD cache.
Default value: True (indicating that the RBD cache is enabled)
Recommended value: True
rbd_cache_size
Specifies the RBD cache size (in bytes).
Default value: 33554432
Recommended value: 335544320
rbd_cache_max_dirty
Specifies the maximum number of dirty bytes allowed when the cache is set to the writeback mode. If the value is 0, the cache is set to the writethrough mode.
Default value: 25165824
Recommended value: 134217728
rbd_cache_max_dirty_age
Specifies the duration (in seconds) for which the dirty data is stored in the cache before being flushed to the drives.
Default value: 1
Recommended value: 30
rbd_cache_writethrough_until_flush
This parameter is used to ensure compatibility with the VirtIO driver earlier than Linux-2.6.32. It allows data to be written back when no flush request is sent. If this parameter is set to True, librbd processes I/Os in writethrough mode, and switches to the writeback mode only when the first flush request is received.
Default value: True
Recommended value: False
rbd_cache_max_dirty_object
Specifies the maximum number of objects. The default value is 0, which indicates that the number of objects is calculated based on the RBD cache size. By default, librbd logically splits the drive image in a unit of 4 MB.
Each chunk object is abstracted as an object. librbd manages the cache objects. You can increase the value of this parameter to improve the performance.
Default value: 0
Recommended value: 2
rbd_cache_target_dirty
Specifies the dirty data size that triggers writeback. The value cannot exceed the value of rbd_cache_max_dirty.
Default value: 16777216
Recommended value: 235544320
Optimizing the PG Distribution
- Purpose
Adjust the number of PGs on each OSD to balance the load on each OSD.
- Procedure
By default, Ceph allocates eight PGs/PGPs to each storage pool. When creating a storage pool, run the ceph osd pool create {pool-name} {pg-num} {pgp-num} command to specify the number of PGs/PGPs, or run the ceph osd pool set {pool_name} pg_num {pg-num} and ceph osd pool set {pool_name} pgp_num {pgp-num} command to change the number of PGs/PGPs created in a storage pool. After the modification, run the ceph osd pool get {pool_name} pg_num/pgp_num command to check the number of PGs/PGPs in the storage pool.
The default value of ceph balancer mode is none. You can run the ceph balancer mode upmap command to change it to upmap. The Ceph balancer function is disabled by default. You can run the ceph balancer on/off command is used to enable or disable the Ceph balancer function.
Table 3 describes the PG distribution parameters.
Table 3 PG distribution parameters Parameter
Description
Suggestion
pg_num
Total PGs = (Total number of OSDs x 100)/Maximum replication count
Round up the result to the nearest integer power of 2.
Default value: 8
Symptom: A warning is displayed if the number of PGs is insufficient.
Suggestion: Calculate the value based on the formula.
pgp_num
Sets the number of PGPs to be the same as that of PGs.
Default value: 8
Symptom: It is recommended that the number of PGPs be the same as the number of PGs.
Suggestion: Calculate the value based on the formula.
ceph_balancer_mode
Enables the balancer plugin and sets the plugin mode to upmap.
Default value: none
Symptom: If the number of PGs is unbalanced, some OSDs may be overloaded and become bottlenecks.
Recommended value: upmap
- The number of PGs carried by each OSD must be the same or close. Otherwise, some OSDs may be overloaded and become bottlenecks. The balancer plugin can be used to optimize the PG distribution. You can run the ceph balancer eval or ceph pg dump command to view the PG distribution.
- Run the eph balancer mode upmap and ceph balancer on commands to automatically balance and optimize Ceph PGs. Ceph adjusts the distribution of a few PGs every 60 seconds. Run the ceph balancer eval or ceph pg dump command to view the PG distribution. If the PG distribution does not change, the distribution is optimal.
- The PG distribution of each OSD affects the load balancing of write data. In addition to optimizing the number of PGs corresponding to each OSD, the distribution of the primary PGs also needs to be optimized. That is, the primary PGs need to be distributed to each OSD as evenly as possible.
Binding OSDs and RGWs to CPU Cores
- Purpose
Bind the OSD and RGW processes to fixed CPU cores to prevent certain CPU cores from being overloaded.
- Procedure
When NIC software interrupts and Ceph processes share CPUs under heavy network load, certain CPUs may be overloaded and become bottlenecks, compromising the Ceph cluster performance. To solve the problem, bind the NIC software interrupts and Ceph processes to different CPU cores. Table 4 describes the parameters to be modified.
Table 4 Binding OSDs and RGWs to CPU cores Parameter
Description
Suggestion
osd.[N]
Binds the osd.n daemon process to a specified idle NUMA node, which is a node other than the nodes that process NIC software interrupts.
Default value: none
Suggestion: Bind the osd.N daemon process to specified CPU cores that do not process NIC software interrupts to prevent the CPU from becoming a bottleneck.
rgw.[N]
Binds the RGW daemon process to a specified idle NUMA node, which is a node other than the nodes that process NIC software interrupts.
Default value: none
Suggestion: Bind the rgw.N daemon process to CPU cores that do not process NIC software interrupts to prevent the CPU from becoming a bottleneck.
The Ceph OSD/RGW daemon process and NIC software interrupt process must run on different CPU cores. Otherwise, CPU bottlenecks may occur when the network load is heavy.
Run the following commands on all Ceph nodes to bind the CPU cores:
1 2
for i in `ps -ef | grep rgw | grep -v grep | awk '{print $2}'`; do taskset -pc 4-47 $i; done for i in `ps -ef | grep osd | grep -v grep | awk '{print $2}'`; do taskset -pc 4-47 $i; done
Optimizing the Network Performance describes how to bind NIC software interrupts to the CPU core of the NUMA node to which the NIC belongs. When the network load is heavy, the usage of the CPU core bound to the software interrupts is high. Therefore, you are advised to set osd_numa_node to a NUMA node different from that of the NIC. For example, run the cat /sys/class/net/<Port Name>/device/numa_node command to query the NUMA node of the NIC. If the NIC belongs to NUMA node 2, set osd_numa_node = 0 or osd_numa_node = 1 to prevent the OSD and NIC software interrupt from using the same CPU core. The core binding of the RGW is similar to that of the OSD. After finding an idle NUMA node, you can run the lscpu command to query the ID of the CPU core that corresponds to the NUMA node. In the preceding command lines, 4-47 indicates the idle CPU cores of the node. Change the value as required.