Introduction

This document describes how to develop vLLM on Kunpeng servers, including source code build, installation, and basic import verification.

The unique PagedAttention mechanism of vLLM introduces operating system virtual memory management to KV cache. Combined with continuous batch processing and other technologies, vLLM achieves a 3 to 24-fold increase in throughput in the Hugging Face Transformers benchmark test, while reducing GPU memory usage by more than 40%. Deploying vLLM on Kunpeng servers leverages the Arm64 architecture compatibility to achieve smooth migration from the x86 platform without any code changes. Additionally, the Kunpeng DevKit and vLLM-Ascend plugin can be used to deeply adapt to Ascend AI processors and implement operator-level fusion and optimization.