Performance Tuning for 10-Billion-Scale Object Storage

Hardware Configuration

Configuring NVMe

• Purpose

  Adjust the NVMe space of the OSD DB partition to enable OSDs to carry more metadata. This prevents metadata overflow in the scenario with 10 billion objects and read/write performance deterioration.

• Procedure

  Each OSD can be allocated with 400 GB (14.6 TB/36) space, 100 GB as the bcache drive and 300 GB as the DB/WAL space. With metadata compression, each OSD can carry more than 800 GB metadata.

Enabling zstd Compression

1. Compile and install the KSAL zstd algorithm. For details, see "Compiling and Installing the KSAL zstd Algorithm Package" in Kunpeng BoostKit for SDS KSAL Developer Guide.
2. Obtain Ceph 14.2.8 source code and save it to the /home directory.
3. Obtain the patch for enabling zstd for RocksDB in Ceph and save it to the /home directory. Download address: https://gitee.com/kunpengcompute/zstd/releases/download/ksal_zstd/ceph-ksal-zstd-plugin.patch
4. Go to the /home directory, decompress the Ceph source package, go to the Ceph source code directory, and apply the patch.

   cd /home && tar -zxvf ceph-14.2.8.tar.gz && cd ceph-14.2.8
   patch -p1 < ../ceph-ksal-zstd-plugin.patch

5. Recompile Ceph. For details, see Ceph 14.2.8 Compilation Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).
   

   Before compiling Ceph, ensure that the KSAL zstd package generated in step 1 has been installed.

6. Install Ceph. For details, see "Installing Ceph" and "Verifying Ceph" in Ceph Object Storage Deployment Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).
7. Modify the ceph.conf file to configure zstd compression.
   

   Complete this step before deploying OSDs.

   [osd]
   bluestore_rocksdb_options="compression=kZSTD,compression_per_level=kNoCompression:kNoCompression:kZSTD:kZSTD:kZSTD,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2"

Adjusting Bcache Parameters

• Purpose

  If SSDs are used to store DB and bcache (SSDs + HDDs) is used to store Data, adjust parameters of the bcache devices and bcache software to improve basic write performance.

• Procedure

  Table 1 describes the parameters to be adjusted.

  Table 1 Bcache parameter adjustment

  Parameter	Description	Tuning Suggestion
  Device parameters	Hardware parameters of cache and data drives used by bcache	Download and install the I/O passthrough tool. For details, see "Installing the Software" in I/O Passthrough Tool User Guide.
  Software parameters	Parameters dynamically specified when the bcache software is running	Run the following script: 1 2 3 4 5 6 7 8 9 10 #!/bin/bash for f in `ls -d /sys/block/bcache*` do echo writeback > $f/bcache/cache_mode echo 0 > $f/bcache/cache/congested_read_threshold_us echo 2000000 > $f/bcache/cache/congested_write_threshold_us echo 0 > $f/bcache/readahead echo 0 > $f/bcache/sequential_cutoff done

Configuring RGW Parameters

• Purpose

  Adjust the number of object gateway shards based on the number of objects to prevent an index shard from recording too many key-value pairs and deteriorating the performance.

• Procedure
  Table 2 describes the parameters to be adjusted.

  Table 2 RGW parameter configuration

  Parameter	Description	Tuning Suggestion
  rgw_dynamic_resharding	Dynamic resharding option	Default value: true Suggestion: Add the following content to the [global] label in the ceph.conf file to disable dynamic resharding: 1 rgw_dynamic_resharding = false
  bucket_index_max_shards	Number of bucket index shards	Default value: 1 Suggestion: After the RGW is started, run the following command to obtain the default configuration: 1 radosgw-admin zonegroup get > default.json Modify the bucket_index_max_shards parameter in the default.json file. In this example, set it to the number of objects in the bucket/200,000. Assume there are 10 billion objects, set the number of shards to 50000 and save the modification. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 { "id": "fe30f363-6c02-4cfc-982f-cf22e1090f34", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "340b719a-c970-4178-8b3a-3b308ed1002c", "zones": [ { "id": "340b719a-c970-4178-8b3a-3b308ed1002c", "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 50000, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "" } Overwrite the original configuration with the new configuration. 1 radosgw-admin zonegroup set < default.json Restart RGW for the configuration to take effect. 1 systemctl restart ceph-radosgw.target