EC Turbo Feature Guide

Introduction

EC Turbo is a Huawei-developed feature library for optimizing the performance of Ceph erasure-coded storage pools. EC Turbo improves the I/O read and write performance for data whose length is within a stripe. This document describes how to enable EC Turbo on Ceph.

Release Description

This feature is released with Kunpeng BoostKit 21.0.0.

Security Hardening Statement

Pay attention to the vulnerabilities reported on the Ceph official website and Ceph GitHub, and fix the vulnerabilities as required.

Environment Requirements

Hardware Compatibility

Item	Specifications
CPU	Kunpeng 920

EC Turbo is available only for Kunpeng processors.

Software Compatibility

Software	Version
OS	openEuler 20.03 LTS SP1
Ceph	Ceph 14.2.8

Currently, EC Turbo adapts only to Ceph 14.2.8. If this feature needs to be used in other versions, you can modify the corresponding version based on open-source patches and APIs of this feature.

Application Scenarios

EC Turbo can be used to optimize the performance of erasure-coded data storage pools.

Ceph data storage pools are classified into two types: replicated and erasure-coded. EC is a data protection method. The original data is divided into K data blocks (data chunks) that form a stripe, and M redundant blocks (coding chunks) are generated after the data chunks are encoded. You can restore the original data by using any K chunks in the K+M chunks. You can store data chunks and coding chunks in different locations.

Environment Preparations

Obtaining the EC Turbo Installation Package

Run the gcc --version command to check the GCC version.
```
gcc --version
```
Download the software package.

Download the software package based on the GCC version.
- If the GCC version is earlier than 4.8.1, download BoostKit-ec_1.2.2_abi-cxx-03.zip.
- If the version is 4.8.1 or later, download BoostKit-ec_1.2.2_abi-cxx-11.zip.
The names of the extracted installation packages are boostkit-ec-1.2.2-1.openeuler.aarch64.rpm and boostkit-sdslog-1.2.2-1.openeuler.aarch64.rpm. The boostkit-sdslog installation package is a dependency of the boostkit-ec installation package. Save the RPM packages to the /home/ec_turbo directory.
Install the RPM packages downloaded in step 2.
```
cd /home/ec_turbo   
yum install boostkit-ec-1.2.2-1.openeuler.aarch64.rpm boostkit-sdslog-1.2.2-1.openeuler.aarch64.rpm  
```
The following table lists the new files and their locations.

File Directory

libkps_ec, libkps_bluestore, libsdslog /usr/lib64

hi_coreutil.h, kps_bluestore.h, sdslog.h, sdslog_param.h /usr/include

logrotate_sdslog.conf /etc

cron_sdslog /etc/cron.d

File	Directory
libkps_ec, libkps_bluestore, libsdslog	/usr/lib64
hi_coreutil.h, kps_bluestore.h, sdslog.h, sdslog_param.h	/usr/include
logrotate_sdslog.conf	/etc
cron_sdslog	/etc/cron.d

Obtaining the Ceph Source Code

Obtain Ceph 14.2.8 source code.
Save the source package to the /home directory on the server and decompress the package.
```
cd /home 
tar -zxvf ceph-14.2.8.tar.gz
```

Applying the Ceph EC Patch

If other SDS features need to be added, install patches by referring to the feature guides.

Obtain ceph-14.2.8-ec_turbo-release.patch and save it to /home/ceph-14.2.8.
Back up the original ceph.spec file. A new ceph.spec file will be generated when you apply the patch in the next step.
```
cd /home/ceph-14.2.8 
cp ceph.spec ceph.spec.bak
```
Apply the patch downloaded in step 1.
```
git apply --check ceph-14.2.8-ec_turbo-release.patch    
patch -p1 < ceph-14.2.8-ec_turbo-release.patch  
```
When applying the patch, src/isa-l-related information in the following figure is displayed. You can ignore the information. This problem can be solved by upgrading isa-l. For details, see the next section.

Building the liboath-devel Package

The openEuler software repository does not include the liboath-devel package. You need to build one.

Download related code and patches.

cd /home/oath  
wget --no-check-certificate https://gitee.com/src-openeuler/oath-toolkit/repository/archive/openEuler-21.03-20210330.zip  
unzip openEuler-21.03-20210330.zip    
cd oath-toolkit

Install dependencies.

yum install gtk-doc pam-devel rpmdevtools  
rpmdev-setuptree

Save related files to the /root/rpmbuild/SOURCES directory.

mv /home/oath-toolkit/0001-oath-toolkit-2.6.5-lockfile.patch oath-toolkit-2.6.5.tar.gz /root/rpmbuild/SOURCES/

Build an RPM package.

cd /root/oath-toolkit
rpmbuild -bb oath-toolkit.spec

Use the RPM package as the local Yum repository.

mkdir -p /home/rpm/oath
cp -r /root/rpmbuild/RPMS/*  /home/rpm/oath/
yum install -y createrepo
cd  /home/rpm/oath
createrepo ./

Configure the repository file.
1. Open the local.repo file.
```
vi /etc/yum.repos.d/local.repo
```
2. Press i to enter the insert mode and add the following content to the file:
```
[local-oath]
name=local-oath
baseurl=file:///home/rpm/oath
enabled=1
gpgcheck=0
priority=1
```
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
Install liboath dependencies.
```
yum install liboath liboath-devel -y
```

Compiling and Deploying Ceph

Preparing the Compilation Environment

Currently, Ceph cannot be directly compiled on openEuler. You need to perform the following steps for adaptation.

Return to the ceph-14.2.8 directory.
```
cd /home/ceph-14.2.8  
```
Open the install-deps.sh file.
```
vim install-deps.sh
```
Add openEuler to the file.

Create libkps_bluestore.so and libkps_ec.so.

cd /usr/lib64/ 
ln -snf libkps_bluestore.so.1.0.0 libkps_bluestore.so

Compiling and Verifying Ceph

Preparing the Compilation Environment

Open the yum.conf file.
```
vim /etc/yum.conf
```
Press i to enter the insert mode and add sslverify=false and deltarpm=0.
Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.

Compiling Software

Upgrade isa-l.

The version of isa-l contained in the Ceph 14.2.8 source code is early. Upgrade isa-l.
1. Go to the src directory.
```
cd /home/ceph-14.2.8/src
```
2. Back up the original isa-l and obtain the latest isa-l 2.29 source code.
```
mv isa-l isa-l.bak
wget https://github.com/intel/isa-l/archive/refs/tags/v2.29.0.tar.gz --no-check-certificate
tar -xzvf v2.29.0.tar.gz
mv isa-l-2.29.0 isa-l
```
3. Modify some isa-l source code to adapt to AArch64.
  
  For details about the modification, visit GitHub.
  
  In the file, the red lines are to be deleted, and the green lines are to be added and modified.
  
  After the upgrade is complete, the aarch64 directory exists in isa-l/erasure_code/.

Install dependencies.

cd /home/ceph-14.2.8/ 
sh install-deps.sh

Modify the do_cmake.sh file.
1. Open the file.
```
vim do_cmake.sh
```
2. Press i to enter the insert mode and modify the file as follows:
```
${CMAKE} -DCMAKE_BUILD_TYPE=RelWithDebInfo $ARGS "$@" .. || exit 1
```
  Comment out git submodule update --init --recursive to prevent isa-l from being rolled back to the old version when building an RPM package.
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
Execute do_cmake.sh.
```
sh do_cmake.sh    
```
Start the compilation.

The compilation requires GCC 7 or later. Make the compilation environment ready. {number} in the following commands indicates the number of jobs during compilation. Generally, a larger value indicates a faster compilation speed. However, the value cannot exceed the number of CPU cores.
```
cd /home/ceph-14.2.8/build 
make -j{number}
```
Perform a unit test.
```
ctest3 -V -R unittest_erasure_code
```
Delete the build directory.
```
cd /home/ceph-14.2.8/ 
rm -rf build
```

Creating a Ceph RPM Package

Install rpmbuild.
```
yum install rpmdevtools -y  
rpmdev-setuptree   
```
If you perform the compilation as the root user, an rpmbuild directory is generated in the /root directory. The compilation occupies 20 GB to 30 GB of space. If the /root directory has a smaller space, relocate rpmbuild to another directory, for example, /home.
Modify the .rpmmacros file.
1. Open the file.
```
vim /root/.rpmmacros   
```
2. Press i to enter the insert mode and set %_topdir to /home/rpmbuild.
3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
Run the rpmbuild installation command again.
```
rpmdev-setuptree
```
Copy the ceph.spec file.
```
cp ceph.spec /home/rpmbuild/SPECS/
```

Copy the source package.

cd /home/
tar -cjvf ceph-14.2.8.tar.bz2 ceph-14.2.8
cp ceph-14.2.8.tar.bz2 /home/rpmbuild/SOURCES/

Build RPM packages.

Move performance.sh to the /home directory.

mv /etc/profile.d/performance.sh /home/
source /etc/profile

Build RPM packages.

rpmbuild -bb /home/rpmbuild/SPECS/ceph.spec

Restore performance.sh.

mv /home/performance.sh /etc/profile.d/   
source /etc/profile

Deploying a Ceph Cluster

Create a local repository.
1. Create a /home/ecturbo_rpm directory and copy the created RPM packages to the directory.
```
mkdir /home/ecturbo_rpm 
cp /home/rpmbuild/RPMS/aarch64/* /home/ecturbo_rpm   
cp /home/rpmbuild/RPMS/noarch/* /home/ecturbo_rpm
```
2. Go to /home/ecturbo_rpm and generate repodata.
```
cd /home/ecturbo_rpm
createrepo ./
```
3. Go to /etc/yum.repos.d and create a my_ceph_local.repo file.
```
cd /etc/yum.repos.d
vim my_ceph_local.repo
```
4. Press i to enter the insert mode and edit the my_ceph_local.repo file. Add the information of Ceph and oath.
```
[myceph]
name=myceph
baseurl=file:///home/ecturbo_rpm
enabled=1
gpgcheck=0
    
[local-oath]
name=local-oath
baseurl=file:///home/rpm/oath
enabled=1
gpgcheck=0
```
5. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
6. Update the Yum cache.
```
yum clean all
yum makecache
```
Install Ceph.

For details, see Installing the Ceph Software in the Ceph Block Storage Deployment Guide (CentOS 7.6 & openEuler 20.03).
- In the deployment guide, the configured Ceph repository is an official Ceph repository, which is an RPM package that does not contain the EC Turbo feature. Therefore, you need to configure Ceph using the local repository.
- Before installing Ceph, run the yum list | grep ceph command to ensure that Ceph is installed from the local repository. In the command, myceph is the name of the repository from which Ceph components are installed.
- When installing Ceph, you may need to install other third-party software. Ensure that the network connection is normal.

Usage Guide

Enabling the EC Turbo Feature

Enable the EC Turbo feature.

For details about how to create an EC storage pool, see Creating a Storage Pool in the Ceph Object Storage Deployment Guide (CentOS 7.6 & openEuler 20.03).
- The stripe_unit value of the EC data pool is 256 KB. When k=4 and m=2, the stripe size (stripe_width) is 1 MB. EC Turbo optimizes I/Os smaller than 1 MB.
- EC Turbo uses the isa-l erasure code library. When creating an EC profile, enable the isa-l plugin, that is, add the plugin=isa parameter. In addition, specify stripe_unit=256KB. The command for creating an EC profile cannot be the command in the reference document. The correct command for creating an EC profile is:
```
ceph osd erasure-code-profile set myprofile k=4 m=2 crush-failure-domain=osd crush-device-class=hdd plugin=isa stripe_unit=256K  
```
  If the number of nodes in the cluster is greater than or equal to 6 (k + m), you can set crush-failure-domain to host.
Add the following items to [global] in the ceph.conf file. These parameters can be modified before or after the pool is created.
```
osd_ec_partial_read = true
osd_ec_partial_write = true
osd_ec_partial_update = true
osd_ec_zero_opt = true
bluestore_kpsallocator_enable = true
```
- The partial read option conflicts with the fast read option. To enable osd_ec_partial_read, check that osd_pool_default_ec_fast_read is disabled (disabled by default). Otherwise, partial read does not take effect.
- The default values of the preceding five parameters are true. If you do not add these parameters to ceph.conf, the system uses the default values. To disable the EC Turbo feature, change the values of these parameters in ceph.conf to false.
- Run the ceph daemon osd.<OSD_number> config show | grep -E 'osd_ec_partial_read|osd_ec_partial_write|osd_ec_partial_update|osd_ec_zero_opt|bluestore_kpsallocator_enable' command to check whether the parameters take effect.
Configure EC Turbo log parameters kpsec_log_fullpath, kpsec_log_level, and kpsec_log_memlogsize.

These parameters can be defined in the [global] section of ceph.conf. If you do not define these parameters, the system uses the default values.

The following lists the default values of these parameters.
```
[global]
......
kpsec_log_fullpath = /var/log/ceph/
kpsec_log_level = CRITICAL/DEBUG
kpsec_log_memlogsize = 100
......
```
- The main services of EC Turbo are encapsulated in the libkps_ec dynamic library. This dynamic library does not use the Ceph log system. Therefore, configure logging for it.
- Generally, Ceph logs are stored in /var/log/ceph. If another directory is specified, ensure that the Ceph user can read and write the directory.
- Log levels are classified into the file log level and memory log level, which must be defined separately. Generally, the content recorded in memory logs is more detailed than that recorded in file logs. The log level must be selected from [CRITICAL=0, ERROR=1, WARNING=2, INFORMATION=3, DEBUG=4]. The value of mem_log_size ranges from 100 to 1000.
- If you modify kpsec log parameters when the OSD process is running, restart the OSD process for the modification to take effect.
- If it is inconvenient to restart the OSD process, run the ceph tell osd.<OSD_number> injectargs --<parameter_name>=<parameter_value> command to modify the parameter online, and then run the ceph tell osd.<OSD_number> reload_kps_conf command to reload kpsec log parameters.
Push ceph.conf to each node.
```
ceph-deploy --overwrite-conf admin {node1} {node2}
```
{nodeX} is the name of a Ceph node. You can run the command to push ceph.conf to multiple nodes in the cluster at the same time. After the push is complete, restart the cluster.
Verify that EC Turbo takes effect.
1. Change the log level of ceph-osd to 20/20. For example, change the log level of OSD 3:
```
ceph daemon osd.3 config set debug_osd 20/20
```
2. Perform data read and write in the EC pool. Then view the ceph-osd log numbered 3. If the log contains partial_write=1 partial_update=1, EC Turbo has taken effect.
3. Change the log level of ceph-osd to the default value.
```
ceph daemon osd.3 config set debug_osd 0/1
```

Managing Logs

Use logrotate and crontab to manage logs.

logrotate is a common log management program on Linux, and crontab is a time-based task management system.

/etc/logrotate_sdslog.conf consists log file management parameters of EC Turbo. This file is installed in a specified directory when EC Turbo is installed. The file content is as follows:

/var/log/ceph/kpsec*.log {
    rotate 30
    daily
    compress
    dateext
    dateformat.%Y%m%d.%s
    size=100M
    missingok
    su ceph ceph
    lastaction
        /usr/bin/chmod 440 /var/log/ceph/kpsec*.gz
        /usr/bin/chmod 440 /var/log/ceph/kpsec*.log-*
        /usr/bin/chmod 440 /var/log/ceph/kpsec*.log.*
    endscript
}

/var/log/sdslog*.log {
    rotate 30
    daily
    compress 
    dateext
    dateformat.%Y%m%d.%s
    size=100M
    missingok
    su ceph ceph
    lastaction
        /usr/bin/chmod 440 /var/log/sdslog*.gz
        /usr/bin/chmod 440 /var/log/sdslog*.log-*
        /usr/bin/chmod 440 /var/log/sdslog*.log.*
    endscript
}

crontab can run a specified task at a fixed time, on a fixed date, or at a fixed interval. Add a task to crontab for logrotate to be executed every minute.

Add the cron_sdslog file to the /etc/cron.d/ directory. The file content is as follows:
```
*/1 * * * * root /usr/sbin/logrotate /etc/logrotate_sdslog.conf
```
Change the permission on cron_sdslog to 600.

Troubleshooting

Dependencies Missing During Ceph Compilation

Symptom

During Ceph compilation for enabling the EC Turbo feature, the message Could NOT find LTTngUST (missing: LTTNGUST_LIBRARIES LTTNGUST_INCLUDE_DIRS) is displayed.

Key Process and Cause Analysis

The open-source third-party dependency packages are missing in the environment.

Conclusion and Solution

Install the required dependencies based on the error message.

yum install -y lttng-ust-devel keyutils-libs-devel openldap-devel leveldb-devel snappy-devel lz4-devel curl-devel expat-devel openssl-devel libbabeltrace-devel librabbitmq-devel rdma-core-devel python2-Cython python3-Cython

ceph-deploy mon create-initial Initialization Failure

Symptom

ceph-deploy mon create-initial initialization fails.

Key Process and Cause Analysis

None

Conclusion and Solution

Add a Ceph user group.
```
groupadd ceph
```
Add a user.
```
useradd ceph -g ceph
```
Reinstall Ceph.

`no module named werkzeug.serving` Reported After Ceph Status Is Queried

Symptom

no module named werkzeug.serving is reported after the Ceph status is queried.

Key Process and Cause Analysis

None

Conclusion and Solution

Install werkzeug.

pip install werkzeug

`No module named OpenSSL` Reported When Ceph Status Is Queried

Symptom

No module named OpenSSL is reported when the Ceph status is queried.

Key Process and Cause Analysis

None

Conclusion and Solution

Install OpenSSL.

yum install python2-pyOpenSSL.noarch  
yum install python3-pyOpenSSL.noarch

Failed to Find Dynamic Libraries During do_cmake.sh Execution and Compilation

Symptom

The symptom is similar to the following:

kps_bluestore cannot be found.

Could Not find kps_bluestore (missing: KPS_BLUESTORE_LIBRARIES)
CMake Error at CMakeLists.txt:277 (message):
no libkpsbluestore.so found

lkps_ec cannot be found.

cannot find -lkps_ec
collect2: error: ld returned 1 exit status

Key Process and Cause Analysis

None

Conclusion and Solution

Create soft links.

ln -snf /usr/lib64/libkps_bluestore.so.1.0.0 /usr/lib64/libkps_bluestore.so
ln -snf /usr/lib64/libkps_ec.so.1.2.1 /usr/lib64/libkps_ec.so

Run the compile command again.
```
rm -rf build/ && sh do_cmake.sh
```

Change History

Date	Description
2024-12-30	This is the second official release. Added Ceph all-flash storage support for EC Turbo.
2022-12-17	This is the first official release.

EC Turbo Feature Guide

Introduction

Environment Requirements

Application Scenarios

Environment Preparations

Obtaining the EC Turbo Installation Package

Obtaining the Ceph Source Code

Applying the Ceph EC Patch

Building the liboath-devel Package

Compiling and Deploying Ceph

Preparing the Compilation Environment

Compiling and Verifying Ceph

Creating a Ceph RPM Package

Deploying a Ceph Cluster

Usage Guide

Enabling the EC Turbo Feature

Managing Logs

Troubleshooting

Dependencies Missing During Ceph Compilation

ceph-deploy mon create-initial Initialization Failure

no module named werkzeug.serving Reported After Ceph Status Is Queried

No module named OpenSSL Reported When Ceph Status Is Queried

Failed to Find Dynamic Libraries During do_cmake.sh Execution and Compilation

Change History

`no module named werkzeug.serving` Reported After Ceph Status Is Queried

`No module named OpenSSL` Reported When Ceph Status Is Queried