Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Profile-guided Optimization

The profile-guided optimization (PGO) technology collects program runtime information (profile) through instrumentation. The compiler replaces the profile to guide branch prediction, inline optimization, and function reordering to make more accurate optimization decisions when generating the target program.

The BiSheng Compiler and GCC for openEuler are available. The BiSheng Compiler has been optimized for the openEuler OS, and thus the PGO effect brought by the BiSheng Compiler is slightly better than that of GCC for openEuler. You can select a compiler as required. The following describes the PGO procedure for the two compilers.

PGO for the BiSheng Compiler

  1. Go to the Ceph source code directory.
    1
    		 
    cd /home/ceph-14.2.10
  2. Apply a patch for adaptation.
    1. Obtain the patch and save it to /home/ceph-14.2.10.

      Download

    2. Apply the patch. 
      1
      		 
      patch -p1 < pgo.patch
  3. Download the BiSheng Compiler and set environment variables.
    1. Contact Huawei engineers to download the BiSheng Compiler.
    2. Save the package in the /home directory and decompress it. 
      1
2
      		 
      cd /home 
tar -zxvf BiShengCompiler-4.1.0-aarch64-linux.tar.gz
    3. Set environment variables. 
      1
2
      		 
      export PATH=/home/BiShengCompiler-4.1.0-aarch64-linux/bin:$PATH
export LD_LIBRARY_PATH=/home/BiShengCompiler-4.1.0-aarch64-linux/lib:$LD_LIBRARY_PATH
  4. Obtain ceph-osd after instrumentation compilation.
    1. Go to the ceph-14.2.10 source code directory and create a do_cmake_release_pgo_gen.sh script. 
      1
2
      		 
      cd /home/ceph-14.2.10
vim do_cmake_release_pgo_gen.sh
    2. Enter the following content in the script: 
      1
2
3
4
5
6
7
8
      		 
      #!/bin/bash
set -x
export PATH=/home/BiShengCompiler-4.1.0-aarch64-linux/bin:$PATH
rm -rf build

OPT_FLAGS="-O3 -flto=thin -fuse-ld=lld -Wno-unused-command-line-argument -fprofile-generate=/media/bisheng-pgo -mllvm -inline-threshold=1000 -Wl,-mllvm,-inline-threshold=1000 -Wl,-zmax-page-size=2097152 -mno-outline-atomics"

./do_cmake.sh -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_MANDIR=/usr/share/man -DCMAKE_INSTALL_DOCDIR=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR=/usr/include -DCMAKE_INSTALL_SYSTEMD_SERVICEDIR=/usr/lib/systemd/system -DWITH_MANPAGE=ON -DWITH_PYTHON3=3.7 -DWITH_MGR_DASHBOARD_FRONTEND=OFF -DWITH_PYTHON2=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS_RELEASE="-O3 -DNDEBUG -Wno-unused-command-line-argument" -DCMAKE_CXX_FLAGS_RELEASE="-O3 -DNDEBUG -Wno-unused-command-line-argument" -DCMAKE_AR="`which llvm-ar`" -DCMAKE_RANLIB="`which llvm-ranlib`" -DWITH_TESTS=OFF -DWITH_CEPHFS_JAVA=ON -DWITH_SELINUX=ON -DWITH_LTTNG=ON -DWITH_BABELTRACE=ON -DWITH_OCF=ON -DWITH_BOOST_CONTEXT=ON -DWITH_LIBRADOSSTRIPER=ON -DWITH_RADOSGW_AMQP_ENDPOINT=ON -DWITH_RADOSGW_KAFKA_ENDPOINT=ON -DBOOST_J=64 -DWITH_GRAFANA=ON -DCMAKE_C_FLAGS="$OPT_FLAGS" -DCMAKE_CXX_FLAGS="$OPT_FLAGS" -DWITH_REENTRANT_STRSIGNAL=ON
    3. After the script is executed, go to the build directory. 
      1
2
      		 
      sh do_cmake_release_pgo_gen.sh
cd build
    4. Perform compilation. 
      1
      		 
      make -j
  5. Replace the original ceph-osd with the instrumented one and execute the test case.
    1. Stop the OSD process. 
      1
      		 
      systemctl stop ceph-osd.target
    2. Replace the original ceph-osd in the /media/ directory with the new ceph-osd obtained in step 4 and perform privilege escalation. 
      1
2
      		 
      cp /home/ceph-14.2.10/build/bin/ceph-osd /media
setcap 'CAP_DAC_OVERRIDE+eip CAP_SYS_ADMIN+eip' /media/ceph-osd
    3. Restart the OSD process. 
      1
      		 
      systemctl restart ceph-osd.target
    4. Execute the 4 KB block read/write test case. For details, see Test Procedure.
  6. Collect test case feedback information.
    1. Select an OSD process, for example, osd0, to generate a sampling file. 
      1
2
3
4
      		 
      gdb attach -p $( ps -ef | grep ceph-osd | grep "\-\-id 0" | awk -F" " '{print $2}') 
set height 0
handle SIG36 SIGUSR2 noprint nostop
call (int)exit(0)
    2. After the preceding commands are executed, the default_*_profraw file is generated in /media/bisheng-pgo/.

    3. Preprocess the default_*_profraw file to generate a profraw file. 
      1
2
      		 
      cd /media/bisheng-pgo
/home/BiShengCompiler-4.1.0-aarch64-linux/bin/llvm-profdata merge -output=bisheng.profdata ./*

  7. Recompile ceph-osd based on the feedback information to obtain an optimized ceph-osd.
    1. Go to the ceph-14.2.10 source code directory and create a do_cmake_release_pgo_use.sh script. 
      1
2
      		 
      cd /home/ceph-14.2.10
vim do_cmake_release_pgo_use.sh
    2. Enter the following content in the script: 
      1
2
3
4
5
6
7
8
      		 
      #!/bin/bash
set -x
export PATH=/home/BiShengCompiler-4.1.0-aarch64-linux/bin:$PATH
rm -rf build

OPT_FLAGS="-O3 -flto=thin -fuse-ld=lld -Wno-unused-command-line-argument -fprofile-use=/media/bisheng-pgo/bisheng.profdata -mllvm -inline-threshold=1000 -Wl,-mllvm,-inline-threshold=1000 -Wl,-q -Wl,-zmax-page-size=2097152 -mno-outline-atomics"

./do_cmake.sh -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_MANDIR=/usr/share/man -DCMAKE_INSTALL_DOCDIR=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR=/usr/include -DCMAKE_INSTALL_SYSTEMD_SERVICEDIR=/usr/lib/systemd/system -DWITH_MANPAGE=ON -DWITH_PYTHON3=3.7 -DWITH_MGR_DASHBOARD_FRONTEND=OFF -DWITH_PYTHON2=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS_RELEASE="-O3 -DNDEBUG -Wno-unused-command-line-argument" -DCMAKE_CXX_FLAGS_RELEASE="-O3 -DNDEBUG -Wno-unused-command-line-argument" -DCMAKE_AR="`which llvm-ar`" -DCMAKE_RANLIB="`which llvm-ranlib`" -DWITH_TESTS=OFF -DWITH_CEPHFS_JAVA=ON -DWITH_SELINUX=ON -DWITH_LTTNG=ON -DWITH_BABELTRACE=ON -DWITH_OCF=ON -DWITH_BOOST_CONTEXT=ON -DWITH_LIBRADOSSTRIPER=ON -DWITH_RADOSGW_AMQP_ENDPOINT=ON -DWITH_RADOSGW_KAFKA_ENDPOINT=ON -DBOOST_J=64 -DWITH_GRAFANA=ON -DCMAKE_C_FLAGS="$OPT_FLAGS" -DCMAKE_CXX_FLAGS="$OPT_FLAGS" -DWITH_REENTRANT_STRSIGNAL=ON
    3. After the script is executed, go to the build directory. 
      1
2
      		 
      sh do_cmake_release_pgo_use.sh
cd build
    4. Perform compilation. 
      1
      		 
      make -j
  8. Replace ceph-osd with the optimized one and perform privilege escalation.
    1. Stop the OSD process. 
      1
      		 
      systemctl stop ceph-osd.target
    2. Replace ceph-osd in the /media/ directory with the optimized one obtained in step 7 and perform privilege escalation. 
      1
2
      		 
      cp /home/ceph-14.2.10/build/bin/ceph-osd /media
setcap 'CAP_DAC_OVERRIDE+eip CAP_SYS_ADMIN+eip' /media/ceph-osd
  9. Restart the OSD process in Ceph.
    1
    		 
    systemctl restart ceph-osd.target

PGO for GCC for openEuler

  1. Go to the Ceph source code directory.
    1
    		 
    cd /home/ceph-14.2.10
  2. Apply a patch for adaptation.
    1. Download the patch and save it to /home/ceph-14.2.10.
    2. Apply the patch. 
      1
      		 
      patch -p1 < pgo.patch
  3. Obtain ceph-osd after instrumentation compilation.
    1. Go to the ceph-14.2.10 source code directory and create a do_cmake_release_pgo_gen.sh script. 
      1
2
      		 
      cd /home/ceph-14.2.10
vim do_cmake_release_pgo_gen.sh
    2. Enter the following content in the script: 
      1
2
3
4
5
6
      		 
      #!/bin/bash
set -x
rm -rf build
OPT_FLAGS="-O3 -fprofile-generate=/media/pgo -Wl,-zmax-page-size=2097152"

./do_cmake.sh -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_MANDIR=/usr/share/man -DCMAKE_INSTALL_DOCDIR=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR=/usr/include -DCMAKE_INSTALL_SYSTEMD_SERVICEDIR=/usr/lib/systemd/system -DWITH_MANPAGE=ON -DWITH_PYTHON3=3.7 -DWITH_MGR_DASHBOARD_FRONTEND=OFF -DWITH_PYTHON2=ON -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DWITH_TESTS=OFF -DWITH_CEPHFS_JAVA=ON -DWITH_SELINUX=ON -DWITH_LTTNG=ON -DWITH_BABELTRACE=ON -DWITH_OCF=ON -DWITH_BOOST_CONTEXT=ON -DWITH_LIBRADOSSTRIPER=ON -DWITH_RADOSGW_AMQP_ENDPOINT=ON -DWITH_RADOSGW_KAFKA_ENDPOINT=ON -DBOOST_J=64 -DWITH_GRAFANA=ON -DCMAKE_C_FLAGS="$OPT_FLAGS" -DCMAKE_CXX_FLAGS="$OPT_FLAGS" -DWITH_REENTRANT_STRSIGNAL=ON
    3. After the script is executed, go to the build directory. 
      1
2
      		 
      sh do_cmake_release_pgo_gen.sh
cd build
    4. Perform compilation. 
      1
      		 
      make -j
  4. Replace the original ceph-osd with the instrumented one and execute the test case.
    1. Stop the OSD process. 
      1
      		 
      systemctl stop ceph-osd.target
    2. Replace ceph-osd in the /media/ directory with the instrumented one obtained in step 3 and perform privilege escalation. 
      1
2
      		 
      cp /home/ceph-14.2.10/build/bin/ceph-osd /media
setcap 'CAP_DAC_OVERRIDE+eip CAP_SYS_ADMIN+eip' /media/ceph-osd
    3. Restart the OSD process. 
      1
      		 
      systemctl restart ceph-osd.target
    4. Execute the 4 KB block read/write test case. For details, see Test Procedure.
  5. Collect test case feedback information.
    1. Select an OSD process, for example, osd0, to generate a sampling file. 
      1
2
      		 
      gdb attach -p $(ps -ef | grep ceph-osd | grep "\-\-id 0" | awk -F" " '{print $2}')
call (void) __gcov_exit()
    2. After the preceding commands are executed, the CMakeFiles file is generated in /media/pgo/.

  6. Recompile ceph-osd based on the feedback information to obtain an optimized ceph-osd.
    1. Go to the ceph-14.2.10 source code directory and create a do_cmake_release_pgo_use.sh script. 
      1
2
      		 
      cd /home/ceph-14.2.10
vim do_cmake_release_pgo_use.sh
    2. Enter the following content in the script: 
      1
2
3
4
5
6
      		 
      #!/bin/bash
set -x
rm -rf build
OPT_FLAGS="-O3 -fprofile-use=/media/pgo -fprofile-correction -Wno-error=coverage-mismatch -Wl,-zmax-page-size=2097152"

./do_cmake.sh -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_MANDIR=/usr/share/man -DCMAKE_INSTALL_DOCDIR=/usr/share/doc/ceph -DCMAKE_INSTALL_INCLUDEDIR=/usr/include -DCMAKE_INSTALL_SYSTEMD_SERVICEDIR=/usr/lib/systemd/system -DWITH_MANPAGE=ON -DWITH_PYTHON3=3.7 -DWITH_MGR_DASHBOARD_FRONTEND=OFF -DWITH_PYTHON2=ON -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DWITH_TESTS=OFF -DWITH_CEPHFS_JAVA=ON -DWITH_SELINUX=ON -DWITH_LTTNG=ON -DWITH_BABELTRACE=ON -DWITH_OCF=ON -DWITH_BOOST_CONTEXT=ON -DWITH_LIBRADOSSTRIPER=ON -DWITH_RADOSGW_AMQP_ENDPOINT=ON -DWITH_RADOSGW_KAFKA_ENDPOINT=ON -DBOOST_J=64 -DWITH_GRAFANA=ON -DCMAKE_C_FLAGS="$OPT_FLAGS" -DCMAKE_CXX_FLAGS="$OPT_FLAGS" -DWITH_REENTRANT_STRSIGNAL=ON
    3. After the script is executed, go to the build directory. 
      1
2
      		 
      sh do_cmake_release_pgo_use.sh
cd build
    4. Perform compilation. 
      1
      		 
      make -j
  7. Replace ceph-osd with the optimized one and perform privilege escalation.
    1. Stop the OSD process. 
      1
      		 
      systemctl stop ceph-osd.target
    2. Replace ceph-osd in the /media/ directory with the optimized one obtained in step 6 and perform privilege escalation. 
      1
2
      		 
      cp /home/ceph-14.2.10/build/bin/ceph-osd /media
setcap 'CAP_DAC_OVERRIDE+eip CAP_SYS_ADMIN+eip' /media/ceph-osd
  8. Restart the OSD process in Ceph.
    1
    		 
    systemctl restart ceph-osd.target
Parent topic: EC Turbo Tuning Guide