What's New

1

Added the llama-CPP documentation.

• Released a collection of CPU operator optimization patches based on the official llama.cpp for Kunpeng 950 processors and new Kunpeng 920 processor models, focusing on FP16/FP32/Q8_0 matrix multiplication and fused SDPA attention operators.
• Added the feature description, quick start, release notes, and technical report.

BoostAI Infra llama-CPP Documentation

1

Changed the product name.

Changed "Kunpeng BoostKit for AI" to "Kunpeng BoostKit BoostAI Infra".

Covering the Kunpeng BoostKit BoostAI Infra document architecture

2

Adjusted the document structure.

Added the following document categories:

• Open-source enablement: provides enablement documents for open-source software from upstream communities.
• Acceleration: provides enablement documents for underlying acceleration suite features and optimization patches based on open-source software.
• Tuning guide: provides hardware/software tuning configuration documents.

3

Added the vLLM-ops documentation.

• vLLM-ops enables performance optimization using new Kunpeng 920 processor models and MetaX C500 GPUs based on vLLM 0.11.0 and vLLM-metax 0.11.0-dev.
• Provided optimization patch introduction, API description, and patch enablement documents.

BoostAI Infra vLLM-ops Documentation

4

Added documents for open-source enablement.

Added the developer and installation guides for 16 types of AI software in the ecosystem enablement.

• Developer guide: describes how to develop open-source AI software on Kunpeng servers, including source code build, installation, and development verification.
• Installation guide: describes how to install open-source AI software on Kunpeng servers and verify basic compatibility and functions.

• Dify
• FlashAttention
• LangChain
• LlamaIndex
• NumPy
• Ollama
• OpenClaw
• Paddle Inference
• PaddlePaddle
• PyTorch
• Safetensors
• SGLang
• TensorFlow
• Tokenizers
• Transformers
• vLLM

1

Migrated the CMF to the GitCode open-source platform.

Open-source software packages and documents are provided on GitCode.

BoostAI Infra CMF Documentation

2

Added the data parallel routing plugin vLLM-Router.

• Open-source software packages and documents are provided on GitCode.
• vLLM-Router is a routing plugin based on the vLLM open-source community. It aims to support data parallel deployment and provide high-performance request routing and load balancing capabilities.

BoostAI Infra vLLM-Router Documentation

1

Added the Cache Management Framework (CMF).

CMF is developed based on the Kunpeng hardware platform and consists of a kernel-mode driver and a command line tool. It modifies hardware registers to control the allocation of system resources such as the L2 cache.

BoostAI Infra CMF Documentation

1

Updated the DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 800I A2 Inference Server).

Updated the MindIE Turbo version to 2.0.RC2 and added the model quantization performance tuning methods.

DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 800I A2 Inference Server)

1

Added the DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 800I A2 Inference Server).

Details the deployment procedures for vLLM, vLLM-Ascend, and MindIE Turbo frameworks on Atlas 800I A2 inference servers running on Kunpeng 920 processors, covering both execution and tuning techniques for the DeepSeek 70B model.

DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 800I A2 Inference Server)

2

Added the DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 300I Duo Inference Card).

Details the deployment of the DeepSeek 70B model in an environment with Kunpeng 920 and two Atlas 300I Duo inference cards, along with performance tuning procedures.

DeepSeek 70B Inference Model Tuning Guide (Kunpeng 920 + Atlas 300I Duo Inference Card)