Documentation
Resource
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Expanding Object Storage Capacity

In this example, create 12 RGW instances on each newly added node, that is, add 24 RGW instances in total for the two nodes. The gateway ports are 100037 to 10060, and the gateway names are bucket37 to bucket60.

Adding MONs to New Servers

In the /etc/ceph/ceph.conf file of ceph1, add the IP addresses of ceph4 and ceph5 to mon_initial_members and mon_host.

  1. Modify the ceph.conf file.
    1
2
3
    cd
/etc/ceph/
vim ceph.conf
    1. Change mon_initial_members=ceph1,ceph2,ceph3 to mon_initial_members=ceph1,ceph2,ceph3,ceph4,ceph5.
    2. Change mon_host=192.168.3.156,192.168.3.157,192.168.3.158 to mon_host=192.168.3.156,192.168.3.157,192.168.3.158,192.168.3.197,192.168.3.198.

  2. Push the ceph.conf file from the ceph1 node to other nodes.
    1
    ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3 ceph4 ceph5
  3. Create MONs on ceph1.
    1
    ceph-deploy mon create ceph4 ceph5
  4. Check the MON status.
    1
    ceph mon stat

    If information about the new servers is displayed in the command output, the MONs are created on the new servers.

(Optional) Deleting MONs

Deleting a MON has a great impact on the cluster. Plan the MONs in advance and avoid deleting MONs.

The following describes how to delete MONs from ceph2 and ceph3 as an example. Modify the ceph2 and ceph3 information in the /etc/ceph/ceph.conf file of ceph1 and push the ceph.conf file to other nodes.

  1. Modify ceph.conf.
    1
2
3
    cd
/etc/ceph/
vim ceph.conf

    Change mon_initial_members=ceph1,ceph2,ceph3,ceph4,ceph5 to mon_initial_members=ceph1,ceph4,ceph5.

    Change mon_host=192.168.3.156,192.168.3.157,192.168.3.158,192.168.3.197,192.168.3.198 to mon_host=192.168.3.156,192.168.3.197,192.168.3.198.

  2. Push the ceph.conf file from the ceph1 node to other nodes.
    1
    ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3 ceph4 ceph5
  3. Delete the MONs from ceph2 and ceph3.
    1
    ceph-deploy mon destroy ceph2 ceph3

Modifying the ceph.conf File

  1. In ceph.conf on ceph1, add the port configuration of the RGW instances.
    1
    vim /etc/ceph/ceph.conf

      1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
    [global]
fsid = 4f238985-ad0a-4fc3-944b-da59ea3e65d7
mon_initial_members = ceph1, ceph2, ceph3,ceph4,ceph5
mon_host = 192.168.3.156,192.168.3.157,192.168.3.158, 192.168.3.197, 192.168.3.198
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 192.168.3.0/24
cluster_network = 192.168.3.0/24

[mon]
mon_allow_pool_delete = true

[client.rgw.bucket1]
rgw_frontends = civetweb port=10001
log file = /var/log/ceph/client.rgw.bucket1.log
[client.rgw.bucket2]
rgw_frontends = civetweb port=10002
log file = /var/log/ceph/client.rgw.bucket2.log
[client.rgw.bucket3]
rgw_frontends = civetweb port=10003
log file = /var/log/ceph/client.rgw.bucket3.log
[client.rgw.bucket4]
rgw_frontends = civetweb port=10004
log file = /var/log/ceph/client.rgw.bucket4.log
[client.rgw.bucket5]
rgw_frontends = civetweb port=10005
log file = /var/log/ceph/client.rgw.bucket5.log
[client.rgw.bucket6]
rgw_frontends = civetweb port=10006
log file = /var/log/ceph/client.rgw.bucket6.log
[client.rgw.bucket7]
rgw_frontends = civetweb port=10007
log file = /var/log/ceph/client.rgw.bucket7.log
[client.rgw.bucket8]
rgw_frontends = civetweb port=10008
log file = /var/log/ceph/client.rgw.bucket8.log
[client.rgw.bucket9]
rgw_frontends = civetweb port=10009
log file = /var/log/ceph/client.rgw.bucket9.log
[client.rgw.bucket10]
rgw_frontends = civetweb port=10010
log file = /var/log/ceph/client.rgw.bucket10.log
[client.rgw.bucket11]
rgw_frontends = civetweb port=10011
log file = /var/log/ceph/client.rgw.bucket11.log
[client.rgw.bucket12]
rgw_frontends = civetweb port=10012
log file = /var/log/ceph/client.rgw.bucket12.log
[client.rgw.bucket13]
rgw_frontends = civetweb port=10013
log file = /var/log/ceph/client.rgw.bucket13.log
[client.rgw.bucket14]
rgw_frontends = civetweb port=10014
log file = /var/log/ceph/client.rgw.bucket14.log
[client.rgw.bucket15]
rgw_frontends = civetweb port=10015
log file = /var/log/ceph/client.rgw.bucket15.log
[client.rgw.bucket16]
rgw_frontends = civetweb port=10016
log file = /var/log/ceph/client.rgw.bucket16.log
[client.rgw.bucket17]
rgw_frontends = civetweb port=10017
log file = /var/log/ceph/client.rgw.bucket17.log
[client.rgw.bucket18]
rgw_frontends = civetweb port=10018
log file = /var/log/ceph/client.rgw.bucket18.log
[client.rgw.bucket19]
rgw_frontends = civetweb port=10019
log file = /var/log/ceph/client.rgw.bucket19.log
[client.rgw.bucket20]
rgw_frontends = civetweb port=10020
log file = /var/log/ceph/client.rgw.bucket20.log
[client.rgw.bucket21]
rgw_frontends = civetweb port=10021
log file = /var/log/ceph/client.rgw.bucket21.log
[client.rgw.bucket22]
rgw_frontends = civetweb port=10022
log file = /var/log/ceph/client.rgw.bucket22.log
[client.rgw.bucket23]
rgw_frontends = civetweb port=10023
log file = /var/log/ceph/client.rgw.bucket23.log
[client.rgw.bucket24]
rgw_frontends = civetweb port=10024
log file = /var/log/ceph/client.rgw.bucket24.log
[client.rgw.bucket25]
rgw_frontends = civetweb port=10025
log file = /var/log/ceph/client.rgw.bucket25.log
[client.rgw.bucket26]
rgw_frontends = civetweb port=10026
log file = /var/log/ceph/client.rgw.bucket26.log
[client.rgw.bucket27]
rgw_frontends = civetweb port=10027
log file = /var/log/ceph/client.rgw.bucket27.log
[client.rgw.bucket28]
rgw_frontends = civetweb port=10028
log file = /var/log/ceph/client.rgw.bucket28.log
[client.rgw.bucket29]
rgw_frontends = civetweb port=10029
log file = /var/log/ceph/client.rgw.bucket29.log
[client.rgw.bucket30]
rgw_frontends = civetweb port=10030
log file = /var/log/ceph/client.rgw.bucket30.log
[client.rgw.bucket31]
rgw_frontends = civetweb port=10031
log file = /var/log/ceph/client.rgw.bucket31.log
[client.rgw.bucket32]
rgw_frontends = civetweb port=10032
log file = /var/log/ceph/client.rgw.bucket32.log
[client.rgw.bucket33]
rgw_frontends = civetweb port=10033
log file = /var/log/ceph/client.rgw.bucket33.log
[client.rgw.bucket34]
rgw_frontends = civetweb port=10034
log file = /var/log/ceph/client.rgw.bucket34.log
[client.rgw.bucket35]
rgw_frontends = civetweb port=10035
log file = /var/log/ceph/client.rgw.bucket35.log
[client.rgw.bucket36]
rgw_frontends = civetweb port=10036
log file = /var/log/ceph/client.rgw.bucket36.log
[client.rgw.bucket37]
rgw_frontends = civetweb port=10037
log file = /var/log/ceph/client.rgw.bucket37.log
[client.rgw.bucket38]
rgw_frontends = civetweb port=10038
log file = /var/log/ceph/client.rgw.bucket38.log
[client.rgw.bucket39]
rgw_frontends = civetweb port=10039
log file = /var/log/ceph/client.rgw.bucket39.log
[client.rgw.bucket41]
rgw_frontends = civetweb port=10041
log file = /var/log/ceph/client.rgw.bucket41.log
[client.rgw.bucket42]
rgw_frontends = civetweb port=10042
log file = /var/log/ceph/client.rgw.bucket42.log
[client.rgw.bucket43]
rgw_frontends = civetweb port=10043
log file = /var/log/ceph/client.rgw.bucket43.log
[client.rgw.bucket44]
rgw_frontends = civetweb port=10044
log file = /var/log/ceph/client.rgw.bucket44.log
[client.rgw.bucket45]
rgw_frontends = civetweb port=10045
log file = /var/log/ceph/client.rgw.bucket45.log
[client.rgw.bucket46]
rgw_frontends = civetweb port=10046
log file = /var/log/ceph/client.rgw.bucket46.log
[client.rgw.bucket47]
rgw_frontends = civetweb port=10047
log file = /var/log/ceph/client.rgw.bucket47.log
[client.rgw.bucket48]
rgw_frontends = civetweb port=10048
log file = /var/log/ceph/client.rgw.bucket48.log
[client.rgw.bucket49]
rgw_frontends = civetweb port=10049
log file = /var/log/ceph/client.rgw.bucket49.log
[client.rgw.bucket50]
rgw_frontends = civetweb port=10050
log file = /var/log/ceph/client.rgw.bucket50.log
[client.rgw.bucket51]
rgw_frontends = civetweb port=10051
log file = /var/log/ceph/client.rgw.bucket51.log
[client.rgw.bucket52]
rgw_frontends = civetweb port=10052
log file = /var/log/ceph/client.rgw.bucket52.log
[client.rgw.bucket53]
rgw_frontends = civetweb port=10053
log file = /var/log/ceph/client.rgw.bucket53.log
[client.rgw.bucket54]
rgw_frontends = civetweb port=10054
log file = /var/log/ceph/client.rgw.bucket54.log
[client.rgw.bucket55]
rgw_frontends = civetweb port=10055
log file = /var/log/ceph/client.rgw.bucket55.log
[client.rgw.bucket56]
rgw_frontends = civetweb port=10056
log file = /var/log/ceph/client.rgw.bucket56.log
[client.rgw.bucket57]
rgw_frontends = civetweb port=10057
log file = /var/log/ceph/client.rgw.bucket57.log
[client.rgw.bucket58]
rgw_frontends = civetweb port=10058
log file = /var/log/ceph/client.rgw.bucket58.log
[client.rgw.bucket59]
rgw_frontends = civetweb port=10059
log file = /var/log/ceph/client.rgw.bucket59.log
[client.rgw.bucket60]
rgw_frontends = civetweb port=10060
log file = /var/log/ceph/client.rgw.bucket60.log
  2. Run the following command on ceph1 to synchronize configuration files on all cluster nodes:
    1
    ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3 ceph4 ceph5

Adding RGW Instances

  1. Run the following command on ceph 1 to create RGW instances.
    1
2
    for i in {37..48};do ceph-deploy rgw create ceph4:bucket$i;done
for i in {49..60};do ceph-deploy rgw create ceph5:bucket$i;done
  2. Check whether the 60 RGW processes are online.
    1
    ceph -s
  3. Use cURL or a web browser to log in to the nodes for verification. Information in the following figure is displayed.

    The gateway service is successfully created.

Deploying MGRs

Create MGRs for ceph4 and ceph5.

1
ceph-deploy mgr create ceph4 ceph5

Deploying OSDs

Create OSDs for the new servers (each server has 12 drives).

1
2
3
4
5
6
7
8
for i in {a..l}
do
ceph-deploy osd create ceph4 --data /dev/sd${i}
done
for i in {a..l}
do
ceph-deploy osd create ceph5 --data /dev/sd${i}
done

Configuring Storage Pools

  1. Query the storage pool information.
    1
    ceph osd lspools

  2. Modify pg pgpnum and pgnum.

    The calculation rule for PGs is as follows:

    Total PGs = ((Total_number_of_OSD * 100) / max_replication_count) / pool_count

    Modify pgnum and pgpnum as follows:
    1
2
3
4
    ceph osd pool set default.rgw.buckets.data pg_num 2048
ceph osd pool set default.rgw.buckets.data pgp_num 2048
ceph osd pool set default.rgw.buckets.index pg_num 256
ceph osd pool set default.rgw.buckets.index pgp_num 256

Verifying Capacity Expansion

After capacity expansion, Ceph migrates some PGs from other OSDs to the new OSDs to balance data again.

  1. Check whether the cluster is healthy after data migration is complete.
    1
    ceph -s

  2. Check that the storage capacity of the cluster has increased.
    1
    ceph osd df

Parent topic: Expanding Ceph Capacity
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details