Introduction

This document describes how to install and verify Tokenizers on Kunpeng servers.

Tokenizers completely bypasses the Python Global Interpreter Lock (GIL) restriction and natively supports multi-thread parallel processing of text in batches. This allows it to tokenize 1 GB of text within 20 seconds on server CPUs. It is widely used in large language model (LLM) inference preprocessing and custom word library-based training scenarios. When Tokenizers is deployed on Kunpeng servers, its architecture comprising a Rust core with Python bindings can be directly installed and run on the AArch64 platform (official PyPI has provided AArch64 precompiled wheels). This allows Tokenizers to fully utilize the multi-core parallel architecture of Kunpeng processors to implement linear acceleration for batch text tokenization.