README

Project Introduction

Metadata acceleration is a storage engine performance acceleration feature developed by Huawei and optimized based on RocksDB.

RocksDB is a high-performance, persistent, and embedded key-value storage engine. It is widely used in large-scale data storage and processing, such as Internet services, distributed systems, and data analysis services. Based on RocksDB, the metadata acceleration feature uses a Huawei-developed algorithm to enable Kunpeng acceleration for better storage performance. This feature fits well with the Kunpeng architecture to optimize read and write hotspots, adjust background tasks (data flushing and compaction) based on service loads, and optimize cache logic based on data hotspots.

Directory Structure

├── docs                                    # Project documentation
│   ├── LICENSE                             # Document license
│   └── en                                  # Document directory
│       ├── figures                         # Directory of figures in documents
│       ├── public_sys-resources            # Public resource directory
│       ├── installation_guide.md           # Installation guide
│       └── release_notes.md                # Release notes
├── 6.1.2-optimization.patch                # Optimization patch (version 6.1.2)
├── rocksdb-8.3.3-kae_zstd.patch            # Optimization patch (version 8.3.3)
├── LICENSE                                 # Software license
└── README_EN.md                            # Project introduction

Version Description

For details about the version information, see Release Notes.

Environment Deployment

For environmental requirements, compilation, and installation, refer to the Installation Guide.

Quick Start

The rocksdb-8.3.3-kae_zstd branch is used as an example. For details about how to deploy the environment, see Installation Guide.

1. Modify the test script.

   cd rocksdb/script
   vim test_perf_all.sh

   Change the value of num1 in the first line to 13421772.

   num1=13421772 # Total number of I/Os
   basedir="/mnt/rocksdb_data/test"
   prefix="../test_data/test_perf_cache10%_final"
   mkdir -p $basedir
   ...

2. Test the performance.

   sh test_perf_all.sh

Note:

• The default read/write path is /mnt/rocksdb_data/test. To change the path, edit the second line basedir in the test_perf_all.sh script to the specified path. Before running the script, ensure that the path is available.

• To mount a device to a specified path, run the following commands (replace nvme0n1 as required):

  mkfs.ext4 /dev/nvme0n1
  mount /dev/nvme0n1 /mnt/rocksdb_data/test

Disclaimer

To Users of This Project

• This project is intended solely for debugging and development. You are responsible for any risks and should carefully review the following information:

  • Data processing and deletion: Users are responsible for managing and deleting any data generated while using this tool. You are advised to promptly delete any related data after use to prevent information leaks.
  • Data confidentiality and transmission: Users understand and agree not to share or transmit any data generated by this tool. Neither the tool nor its developers are responsible for any information leaks, data breaches, or other negative consequences.
  • User input security: Users are responsible for the security of any commands they enter and for any risks or losses resulting from improper input. The tool and its developers are not liable for issues caused by incorrect command usage.

• Disclaimer scope: This disclaimer applies to all individuals and entities using this tool. By using the tool, you acknowledge and accept this statement and assume all risks and responsibilities arising from its use. If you do not agree, please stop using the tool immediately.

• Before using this tool, please read and understand the preceding disclaimer. If you have any questions, contact the developer.

To Data Owners

If you do not want your model or dataset to be mentioned in this project, or if you wish to update its description, please submit an issue on GitCode. We will delete or update your description according to your request. Thank you for your understanding and contribution to this project.

License

The code of this project is licensed under the Apache License 2.0. For details, see LICENSE.

The documents of this project are licensed under CC-BY 4.0. For details, see LICENSE.

Contribution Statement

We welcome your contributions to the community. If you have any questions/suggestions or want to provide feedback on feature requirements and bug reports, you can submit issues. For details, see the contribution guideline. You are also welcome to share insights in Discussions. Thank you for your support.