---
title: 开发验证
description: "执行以下命令进行验证。"
url: https://www.hikunpeng.com/document/detail/zh/kunpengai/ecosystemEnable/tokenizers/kunpengtokenizers_16_0006.html
sourcePath: /source/zh/kunpengai/ecosystemEnable/tokenizers/kunpengtokenizers_16_0006.html
indexId: 4e88312373eb371f965b9e9d7839bb64e261fb7f4a3bdc789640fd93e16c413478
---
# 开发验证

执行以下命令进行验证。

```
python3 - <<'PY'
import tokenizers 
from tokenizers import Tokenizer 
from tokenizers.models import WordLevel 
from tokenizers.pre_tokenizers import Whitespace  

print("tokenizers_version=" + tokenizers.__version__) 
tok = Tokenizer(WordLevel({"[UNK]": 0, "hello": 1, "kunpeng": 2, "world": 3}, unk_token="[UNK]")) 
tok.pre_tokenizer = Whitespace() 
out = tok.encode("hello kunpeng world") 
print("ids=" + ",".join(str(i) for i in out.ids)) 
print("tokens=" + ",".join(out.tokens)) 
assert tokenizers.__version__ == "0.22.2"
assert out.ids == [1, 2, 3] 
assert out.tokens == ["hello", "kunpeng", "world"] 
PY
```

预期输出如下信息：

```
tokenizers_version=0.22.2 
ids=1,2,3 
tokens=hello,kunpeng,world
```