Ranking Inference Scenario
Scenario Architecture
KDNN
Based on the microarchitecture features of the Kunpeng processor, Kunpeng Deep Neural Network Library (KDNN) improves the performance of core DNN operators through vectorization, assembly, and algorithm optimization. It can be integrated into the open-source oneDNN library as a plugin to provide complete capabilities. In addition, it supports interfacing with some matrix operators such as TensorFlow Matmul.
KDNN_EXT
KDNN_EXT serves as the extension library of KDNN. It optimizes operators such as softmax and random_choice, and encapsulates them into a Python interface library for specific AI scenarios.
Kunpeng Inference Acceleration Kit KTFOP
KTFOP is an efficient, Huawei-developed TensorFlow operator library. It uses SIMD instructions and multi-core scheduling to accelerate operator processing in CPUs and reduce the usage of CPU computing resources, thereby increasing the overall end-to-end throughput of online inference.
TensorRT-LLM
Based on the open-source TensorRT-LLM, this component focuses on efficient execution in Large Language Model (LLM) inference scenarios. It enhances performance through in-depth operator optimization, memory access optimization, and parameter tuning, thereby boosting inference throughput and reducing latency.
TensorFlow
Kunpeng TensorFlow is a high-performance inference acceleration extension based on open-source TensorFlow. It focuses on efficient execution in search, recommendation, and advertising inference scenarios. It improves throughput and cuts latency for model inference through in-depth enhancements in graph optimization, operators, and runtime.
Kunpeng AI Compiler ANNC
TensorFlow leverages the Accelerated Neural Network Compiler (ANNC) to perform graph-level optimizations, enhancing inference performance in recommendation systems. ANNC provides optimization technologies including computational graph optimization, and generation and integration of high-performance fused operators.