Source Code Compilation and Build

Obtain the Tokenizers 0.22.2 source code, build a wheel from the source package, and install the build product.

1. Obtain the Tokenizers 0.22.2 source code.

   1 2 3

   mkdir -p /tmp/tokenizers-build/dist python3 -m pip download --no-build-isolation --no-binary=:all: --no-deps \ --dest /tmp/tokenizers-build tokenizers==0.22.2

   

   --no-binary=:all: is used to disable the download of the precompiled wheel and ensure that the source package is obtained and local build is performed.

2. Build the wheel from the source package.

   1 2 3

   python3 -m pip wheel --no-build-isolation --no-binary=:all: --no-deps \ --wheel-dir /tmp/tokenizers-build/dist \ /tmp/tokenizers-build/tokenizers-0.22.2.tar.gz

3. Install the running dependencies and local build product.

   1 2	python3 -m pip install "huggingface-hub>=0.16.4,<2.0" python3 -m pip install --no-index --find-links /tmp/tokenizers-build/dist --no-deps tokenizers==0.22.2

   The following is an example of the build product.

   1	tokenizers-0.22.2-cp39-abi3-linux_aarch64.whl