Data Compaction Feature Guide

Introduction

The data compaction algorithm is deployed on an open-source Ceph cluster to eliminate data waste caused by zero padding. In addition, combined with techniques including data encapsulation, space allocation based on block counting, granularity-based traffic steering, batch submission, and batch callback, the data compaction algorithm improves the data reduction ratio and overall system IOPS, which reduces costs and improves performance.

This document describes how to enable the data compaction algorithm on Ceph. The data compaction algorithm consists of an open-source patch and a closed-source RPM package. After integrating the data compaction software into Ceph source code, compile and deploy Ceph. The data compaction function takes effect in the Ceph cluster.

Release Description

This feature is released with Kunpeng BoostKit 21.0.0.

Security Hardening Statement

Pay attention to the vulnerabilities reported on the Ceph official website and Ceph GitHub, and fix the vulnerabilities as required.

Environment Requirements

The data compaction algorithm can be used only on the Kunpeng platform.

Hardware Requirements

Item	Description
CPU model	Kunpeng 920

Software Requirements

Item	Description
OS	CentOS Linux release 7.6.1810
OS	openEuler 20.03 LTS SP1
GCC	GCC 7.3.0
Ceph	Ceph 14.2.8

Integrating the Data Compaction Function into Ceph

Download the Ceph 14.2.8 source package.
Save the source package to the /home directory on the server and decompress it.
```
cd /home
tar zxvf ceph-14.2.8.tar.gz
```
Apply the data compaction plugin.
1. Download ceph-14.2.8-compaction.patch and save it to the /home/ceph-14.2.8 directory.
2. Apply the patch.
```
cd /home/ceph-14.2.8
patch -p2 < ceph-14.2.8-compaction.patch
```
Download the data compaction installation package to the /home/ceph-14.2.8 directory.

Kunpeng community: BoostKit-compaction_1.0.0.zip
Decompress the installation package.
```
cd /home/ceph-14.2.8/
unzip BoostKit-compaction_1.0.0.zip
```
The boostkit-compaction-1.0.0-1.aarch64.rpm package is extracted.

Install the RPM package.

rpm -ivh boostkit-compaction-1.0.0-1.aarch64.rpm

Compiling and Deploying Ceph

Environment Preparation

The operations vary according to the OS. Unless otherwise specified, the operations are the same on the two OSs.

CentOS 7.6

Install the EPEL repository.
```
yum install epel-release -y
```
Install the software collection (SCL).
```
yum -y install centos-release-scl
```

Modify the SCL repository.

vi /etc/yum.repos.d/CentOS-SCLo-scl.repo

Add the following content:

baseurl=http://mirror.centos.org/altarch/7/sclo/$basearch/rh/

Change http in the scl-rh repository file to https.

vi /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

Configure the Yum certificate verification.
```
vi /etc/yum.conf
```
```
sslverify=false
deltarpm=0
```
Update Yum.
```
yum clean all && yum makecache
```
(Optional) Simulate and verify the GCC 7 compilation environment.

The compilation requires GCC 7 or later. If the GCC version meets the requirement, skip this step. If not, run the following commands to simulate the GCC 7 environment.
```
yum -y install devtoolset-7
scl enable devtoolset-7 bash
gcc --version
```
If the GCC version is 7.0 in the command output, the environment is successfully simulated.

openEuler 20.03

Configure the local liboath repository.

Download the liboath source code and patch.

yum install git -y
git config --global http.sslVerify false
git clone https://gitee.com/src-openeuler/oath-toolkit.git

Use Yum to install the dependencies required for the RPM package.

yum install wget rpmdevtools gtk-doc pam-devel xmlsec1-devel libtool libtool-ltdl-devel createrepo cmake -y

Create an rpmbuild directory. Move the patch file and source package to the /root/rpmbuild/SOURCES directory.

rpmdev-setuptree
cd oath-toolkit
mv 0001-oath-toolkit-2.6.5-lockfile.patch /root/rpmbuild/SOURCES
mv oath-toolkit-2.6.5.tar.gz /root/rpmbuild/SOURCES
cp oath-toolkit.spec /root/rpmbuild/SPECS/

Compile the RPM package.

rpmbuild -bb /root/rpmbuild/SPECS/oath-toolkit.spec

Use the compiled RPM package as the local Yum repository.

mkdir -p /home/oath
cp -r /root/rpmbuild/RPMS/*  /home/oath/
cd  /home/oath && createrepo .

Configure the repository file.

vi /etc/yum.repos.d/local.repo

Add the following content to the file:

[local-oath]
name=local-oath
baseurl=file:///home/oath
enabled=1
gpgcheck=0
priority=1

Edit the yum.conf file and set the Yum certificate verification status to false (no verification).
```
vi /etc/yum.conf
```
Add the following content to the end of the file:
```
sslverify=false
deltarpm=0
```

Configure the Huawei proxy to accelerate the download.

mkdir -p ~/.pip
vi ~/.pip/pip.conf

Add the following content:

[global]
timeout = 120
index-url =https://repo.huaweicloud.com/repository/pypi/simple
trusted-host = repo.huaweicloud.com

Download the openEuler repository.

wget -O /etc/yum.repos.d/openEulerOS.repo https://repo.huaweicloud.com/repository/conf/openeuler_aarch64.repo

Install the dependencies required for compiling Ceph source code on the server.

yum install -y java-devel sharutils checkpolicy selinux-policy-devel gperf cryptsetup fuse-devel /
 gperftools-devel libaio-devel libblkid-devel libcurl-devel libudev-devel libxml2-devel /
 libuuid-devel ncurses-devel python-devel valgrind-devel xfsprogs-devel xmlstarlet yasm /
 nss-devel libibverbs-devel openldap-devel CUnit-devel python2-Cython python3-setuptools /
 python-prettytable lttng-ust-devel expat-devel junit boost-random keyutils-libs-devel openssl-devel /
 libcap-ng-devel python-sphinx python2-sphinx python3-sphinx leveldb leveldb-devel snappy /
 snappy-devel lz4 lz4-devel liboath liboath-devel libbabeltrace-devel librabbitmq librabbitmq-devel /
 librdkafka librdkafka-devel libnl3 libnl3-devel rdma-core-devel numactl numactl-devel numactl-libs /
 createrepo openldap-devel rdma-core-devel lz4-devel expat-devel lttng-ust-devel libbabeltrace-devel /
 python3-Cython python2-Cython gperftools-devel bc dnf-plugins-core librabbitmq-devel rpm-build /
 java-1.8.0-openjdk-devel

Regenerate an rpmbuild directory in /home.
1. Run the rpmbuild installation command.
```
rpmdev-setuptree
```
2. Modify the .rpmmacros file.
```
vi /root/.rpmmacros
```
  Change %_topdir to /home/rpmbuild.
3. Run the rpmbuild installation command again.
```
rpmdev-setuptree
```

Compiling and Verifying Ceph

CentOS 7.6

Modify the ceph.spec file.
```
cd /home/ceph-14.2.8/
vi ceph.spec.in
```
Change the SciPy version to python36-scipy.

Modify dashboard/requirements.txt.

vi /home/ceph-14.2.8/src/pybind/mgr/dashboard/requirements.txt

Comment out pyopenssl.

PyJWT==1.6.4
#pyopenssl==17.5.0
pytest==3.3.2

Install the dependencies.

yum -y install epel-release
yum -y install python36-scipy.aarch64
cd /home/ceph-14.2.8/
sh install-deps.sh

Start the compilation.
```
sh do_cmake.sh
cd build
make -j 48
```
Perform a unit test.
```
ctest3 -V -R unittest_compression
```
Delete the build directory.
```
cd /home/ceph-14.2.8/
rm -rf build
```

openEuler 20.03

Use EPEL to install the missing dependencies in openEuler.
1. Configure the EPEL repository.
```
vi /etc/yum.repos.d/epel.repo
```
  Add the following content:
```
[epel]
name=epel
baseurl=https://repo.huaweicloud.com/epel/7/aarch64/
enabled=1
gpgcheck=0
priority=1
```
2. Use the EPEL repository to install dependencies.
```
yum install python-routes python-tox -y
```
3. Delete the EPEL repository.
```
rm -rf /etc/yum.repos.d/epel.repo
```
  The EPEL repository must be deleted. Otherwise, an RPM package that conflicts with openEuler will be downloaded from the EPEL repository in subsequent steps.
Modify the Ceph-related code to make it compatible with openEuler.
```
cd /home/ceph-14.2.8/
```
1. Add openEuler to the install-deps.sh file.
```
vim install-deps.sh
```
2. Modify the ceph.spec.in file.
```
sed -i 's#%if 0%{?fedora} || 0%{?rhel}#%if 0%{?fedora} || 0%{?rhel} || 0%{?openEuler}#' ceph.spec.in
```
3. Modify the ceph.spec file.
```
vim ceph.spec
```
  1. Add the following content to the beginning of the file:
```
%define _binaries_in_noarch_packages_terminate_build 0
```
  2. Configure CEPH_MFLAGS_JOBS="-j48" to accelerate the compilation of the openEuler version.

Install and compile the dependency.

Install the dependency.

cd /home/ceph-14.2.8/
sh install-deps.sh

Start the compilation.
```
sh do_cmake.sh
cd build
make -j 48
```
Perform a unit test.
```
ctest3 -V -R unittest_compression
```

Generating the RPM Package of the Data Compaction Algorithm

CentOS 7.6

Change the default build mode in do_cmake.sh to the high-performance mode.
```
vi do_cmake.sh
```
Compress the ceph-14.2.8 directory into a tar.bz2 package.
```
cd /home
tar -cjvf ceph-14.2.8.tar.bz2 ceph-14.2.8
```

Copy the ceph.spec file to the SPECS directory.

cp ceph-14.2.8/ceph.spec /home/rpmbuild/SPECS/

Save the package to the SOURCES directory.

cp ceph-14.2.8.tar.bz2 /home/rpmbuild/SOURCES/

Add fields to the beginning of the ceph.spec file.

vi /home/rpmbuild/SPECS/ceph.spec

Add the following fields:

%define _binaries_in_noarch_packages_terminate_build 0

Build RPM packages.
```
rpmbuild -bb /home/rpmbuild/SPECS/ceph.spec
```
The build task takes 20 to 30 minutes. After the build is complete, the aarch64 and noarch directories are generated in the /home/rpmbuild/RPMS directory, which contain Ceph-related RPM packages.

openEuler 20.03

Delete the build directory.
```
cd /home/ceph-14.2.8/
rm -rf build
```
Change the default build mode in do_cmake.sh to the high-performance mode.
```
vi do_cmake.sh
```
Go back to the parent directory and compress the ceph-14.2.8 directory into a tar.bz2 package.
```
cd /home
tar -cjvf ceph-14.2.8.tar.bz2 ceph-14.2.8
```

Copy the ceph.spec file to the SPECS directory.

cp ceph-14.2.8/ceph.spec /home/rpmbuild/SPECS/

Save the package to the SOURCES directory.

cp ceph-14.2.8.tar.bz2 /home/rpmbuild/SOURCES/

Build RPM packages.
1. Remove and back up /etc/profile.d/performance.sh to accelerate the compilation.
```
mv /etc/profile.d/performance.sh /home/
```
2. Start a new terminal and run the rpmbuild command.
```
unset GOMP_CPU_AFFINITY
rpmbuild -bb /home/rpmbuild/SPECS/ceph.spec
```
  The build task takes 20 to 30 minutes. After the build is complete, the aarch64 and noarch directories are generated in the /home/rpmbuild/RPMS directory, which contain Ceph-related RPM packages.

Deploying a Ceph Cluster

Create a local repository.

yum -y install createrepo
mkdir /home/ceph-compaction
cd /home/ceph-compaction
cp /home/rpmbuild/RPMS/aarch64/*rpm ./
createrepo ./
cd /etc/yum.repos.d/
vi ceph-local.repo

[local]
name=local
baseurl=file:///home/ceph-compaction
enable=1
gpgcheck=0
[Ceph-noarch]
name = Ceph noarch packages
baseurl = http://download.ceph.com/rpm-nautilus/el7/noarch
enabled = 1
gpgcheck = 1
type = rpm-md
gpgkey = https://download.ceph.com/keys/release.asc
priority = 1

Deploy MON and MGR nodes.

For details, see the corresponding Ceph deployment guide.
- For block storage, see Installing the Ceph Software, Deploying MON Nodes, and Deploying MGR Nodes in the Ceph Block Storage Deployment Guide.
- For object storage, see Installing the Ceph Software, Deploying MON Nodes, and Deploying MGR Nodes in the Ceph Object Storage Deployment Guide.
- For file storage, see Installing the Ceph Software, Deploying MON Nodes, and Deploying MGR Nodes in the Ceph File Storage Deployment Guide.
In the deployment guides, the configured Ceph repository is an official Ceph repository, which is an RPM package that does not contain the data compaction algorithm plugin. Therefore, you need to configure Ceph using the local repository. The data compaction algorithm supports only Ceph 14.2.8 and needs to be dynamically adjusted during deployment.
Modify the Ceph configuration file ceph.conf.

The product of osd_op_num_shards_hdd and osd_op_num_threads_per_shard_hdd is the number of threads for the OSD process to process I/O requests. The default value is 5*1. You can change the value to 12*2 to deliver the maximum performance of the data compaction algorithm.
- The configuration items provided in this step are applicable only to HDD scenarios.
- The modification can be dynamically adjusted after OSD nodes are deployed.
```
vi /etc/ceph/ceph.conf
```
Change the default number of OSD threads.
```
osd_op_num_shards_hdd = 12
osd_op_num_threads_per_shard_hdd = 2
```
Deploy OSD nodes.

For details, see Deploying OSD Nodes in the Ceph Block Storage Deployment Guide (CentOS 7.6 & openEuler 20.03).

Change History

Date	Description
2021-12-31	This is the first official release.