Introduction

This document describes how to develop Tokenizers on Kunpeng servers, including source code build, installation, and development verification.

Tokenizers completely bypasses the Python Global Interpreter Lock (GIL) restriction and natively supports multi-thread parallel processing of text in batches. This allows it to tokenize 1 GB of text within 20 seconds on server CPUs. It is widely used in large language model (LLM) inference preprocessing and custom word library-based training scenarios. When Tokenizers is deployed on Kunpeng servers, its architecture comprising a Rust core with Python bindings can be directly installed and run on the AArch64 platform (official PyPI has provided AArch64 precompiled wheels). This allows Tokenizers to fully utilize the multi-core parallel architecture of Kunpeng processors to implement linear acceleration for batch text tokenization.

The core implementation of Tokenizers depends on the Rust toolchain. Tokenizers does not depend on GPUs. CUDA or GPU runtime is not required for source code build and function verification.