Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Creating a Workload

COSBench Test Model

The object storage gateway RGW is deployed on one storage node. Four RGW gateway instances with different port numbers are started on one storage node. As described in Installing the Test Tool, the COSBench controller node of the client sends read/write load test commands to the RGW gateways of the storage node.

COSBench Test Process

The COSBench object storage test process consists of three phases: preparation, read/write test, and cleanup. In the preparation phase, an object storage bucket is created and test data is initialized in the bucket. In the read/write test phase, read, write, and mixed read/write tests are performed on the test data in the bucket. In the cleanup phase, objects in the bucket are cleared and the bucket is deleted.

Creating a Load Configuration File

The COSBench load is defined in an XML file. Figure 1 shows the file structure.

Figure 1 Structure of the COSBench load file

The following uses the load of 256 KB as an example to describe the load configuration file of each test process.

  • 256k_prepare.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
        <workstage closuredelay="0" config="" name="createbucket"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw1-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw2-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw3-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw4-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        </workstage> 
 
        <workstage closuredelay="0" config="" name="prepareobject"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare1" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare2" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare3" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare4" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        </workstage> 
    </workflow> 
</workload>
  • 256k_put.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="putobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_get.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="getobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_putget.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="rwobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(1,6);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(7,12);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(13,18);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(19,24);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_clean.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
                <workstage closuredelay="0" config="" name="cleanup"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw1-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw2-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw3-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw4-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        </workstage> 
 
                <workstage closuredelay="0" config="" name="dispose"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw1-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw2-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw3-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw4-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  1. In the previous files:
    • containers indicates the bucket ID.
    • objects indicates the IDs of the objects in each bucket.
    • accesskey and secretkey are created when creating the object storage S3 API user. You can run the radosgw-admin user info --uid=<S3_user_ID> command to query the information.
    • sizes indicates the size of the data block that is read or written each time.
    • endpoint indicates the IP address and port number of the RGW. In the previous configuration files, the IP address 192.168.3.135 and the port numbers are from 10001 to 10004.
  2. The previous configuration files contain the load configuration of the RGW instances corresponding to four different ports on one storage node. If you change the IP address and port number of an endpoint, a new load configuration file is generated for a different RGW. In addition, if you change sizes=c(256)KB to sizes=c(1)MB and sizes=c(4)MB, load configuration files for data block sizes 1 MB and 4 MB are generated.
Parent topic: Test Procedure