EC Metadata Reduction

In Ceph object storage, the data pool is configured to use EC mode to optimize drive utilization and reduce data. The metadata amplification factor corresponds to the number of EC copies. In scenarios with high-ratio EC, metadata amplification becomes significant. To address this, the EC metadata reduction feature separates application object data from metadata by assigning distinct storage classes to buckets and objects within them. This method significantly reduces metadata storage overhead, alleviates the metadata amplification effect caused by multiple EC copies, and improves storage efficiency.

Compiling and Deploying Ceph

If the storage class of a native bucket differs from that of its objects, the multipart upload logic behaves abnormally. The metadata of the first object in each part is stored under the bucket's storage class. Deleting this storage class also removes the metadata, leading to memory leaks. To prevent this issue, ensure that the following patch is applied before building the package.

Obtain Ceph 14.2.8 source code and save it to the /home directory.
Download the metadata reduction patch from Gitee and save it to the /home directory.
Go to the /home directory, decompress the Ceph source package, go to the Ceph source code directory, and apply the patch.
```
cd /home && tar -zxvf ceph-14.2.8.tar.gz && cd ceph-14.2.8
patch -p1 < ../storage_class_bug_fix.patch
```
Re-compile Ceph after applying the patch. For details, see "Compiling and Deploying Ceph" > "Compiling Ceph" in Ceph 14.2.8 Compilation Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).
Install Ceph. For details, see "Installing the Ceph Software" in Ceph Object Storage Deployment Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).
Deploy RGW nodes. For details, see "Deploying RGW Nodes" in Ceph Object Storage Deployment Guide (CentOS 7.6 & openEuler 20.03 & openEuler 22.03).

Deploying Ceph Object Storage

The following describes how to create a storage pool and object class.

When creating a bucket, placement and storage_class must be specified. placement of application objects within the bucket matches that of the bucket itself, but their storage_class may differ. placement and storage_class can be defined during the application object's upload process.
After an RGW gateway is created, there is initially a default-placement with a built-in standard storage class named STANDARD. If placement is not specified, default-placement is used, and if storage_class is not specified, STANDARD is used. The names of index_pool, data_pool, and data_extra_pool under default-placement are fixed.

The bucket is bound to default-placement/head. data_pool corresponding to the head class is named default.rgw.buckets.head and uses a three-replica policy.
Application objects are bound to default-placement/STANDARD. data_pool corresponding to the STANDARD class uses an EC policy.

Create an EC profile.

ceph osd erasure-code-profile set myprofile k=4 m=2 crush-failure-domain=osd

Create a storage pool.

ceph osd pool create default.rgw.buckets.data 2048 2048 erasure myprofile
ceph osd pool create default.rgw.buckets.index 256 256
ceph osd pool create default.rgw.buckets.head 2048 2048
ceph osd pool create default.rgw.buckets.non-ec 256 256
ceph osd pool application enable default.rgw.buckets.data rgw
ceph osd pool application enable default.rgw.buckets.index rgw
ceph osd pool application enable default.rgw.buckets.head rgw
ceph osd pool application enable default.rgw.buckets.non-ec rgw

The two numbers in the storage pool creation command (for example, ceph osd pool create default.rgw.buckets.data 2048 2048 erasure myprofile) respectively correspond to the pg_num and pgp_num parameters of the storage pool.

According to the official Ceph document, the recommended total number of storage pool PGs in a cluster is calculated as follows: (Number of OSDs x 100)/Data redundancy factor. For the replication mode, the data redundancy factor is the number of copies. For the EC mode, the data redundancy factor is the sum of the numbers of data blocks and parity blocks. For example, the data redundancy factor is 3 for the three-replica mode and 6 for the EC 4+2 mode.
Assume that the cluster has three servers and each server has 36 OSDs in the EC 4+2 mode. The total number of OSDs is 108. According to the formula, the PG quantity is 1800. It is recommended that the PG quantity be an integral power of 2. The data volume of default.rgw.buckets.data is much larger than other storage pools and therefore more PGs need to be allocated to this storage pool.
The number of PGs of default.rgw.buckets.head is typically equal to or twice the value of default.rgw.buckets.data.
The number of RADOS objects in default.rgw.buckets.non-ec and default.rgw.buckets.index can not be excessive. You can configure it to be 1/8 of the number of RADOS objects in default.rgw.buckets.data.

Add a storage policy named head to the default zone group.
1
radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class head
- --rgw-zonegroup specifies that the zone group to be operated is default.
- --placement-id specifies that the storage policy to be added is default-placement.
- --storage-class specifies that the storage class of the storage policy is hea.
Add a storage policy named head to the default zone group. The data pool is default.rgw.buckets.head.
1
radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class head --data-pool default.rgw.buckets.head
--data-pool specifies that the data pool of the storage policy is default.rgw.buckets.head.

Creating an RGW Account

Create an RGW account on any node.

radosgw-admin user create --uid="admin" --display-name="admin user"

Query the account information after the account is created.
1
radosgw-admin user info --uid=admin

Creating a Bucket

When creating a bucket, use the following Python script to specify the storage class. Adjust the RGW account information fields (aws_access_key and aws_secret_key) and the RGW process information field (host) according to the actual situation.

import requests
import time
import threading
import sys
from aws_requests_auth.aws_auth import AWSRequestsAuth

def str_generator(n):
    import string
    import random
    chars = string.ascii_letters + string.digits
    return "".join(random.choice(chars) for _ in range(n))

aws_access_key = 'xxxxxxxxxxxxxxxxxxxx'                          # Access key of the RGW account
aws_secret_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'      # Secret key of the RGW account
host = '192.168.xx.xxx:10001'                                    # IP address and listening port of the node where the RGW process resides
region_name = 'default'
service_name = 's3'

aws_auth = AWSRequestsAuth(aws_access_key=aws_access_key,
                           aws_secret_access_key=aws_secret_key,
                           aws_host=host,
                           aws_region=region_name,
                           aws_service=service_name)

headers = {
    "x-amz-storage-class": "STANDARD"                                        # Storage class of the bucket
}


url = f'http://{host}/mybucket'
response = requests.put(url, auth=aws_auth, headers=headers)
if response.status_code == 200: 
    print("Request succeeded")
else: 
    print(f"Request failed. Status code: {response.status_code}")

The x-amz-storage-class field is used to specify the storage class of objects in a bucket. There are three types of common storage classes:

STANDARD: Standard storage class, which is used for frequently accessed data.
INTELLIGENT_TIERING: Intelligent storage tiering, which automatically selects the optimal storage tier based on the access mode.
GLACIER: A low-cost storage class for archive storage, which is suitable for data that is infrequently accessed over the long term.

Parent topic: General-Purpose Storage