The tokenizer has not historically been the bottleneck within ML workflows. Compute-wise, tokenization is light compared to the heavy modeling happening in the rest of the pipeline. Yet, in some cases, it has rapidly become key to accelerating (or slowing down) your machine learning work.
As models become faster and workloads scale, that balance begins to shift. Training on massive datasets, serving many concurrent requests, or repeatedly processing long inputs can put enough pressure on the tokenizer that it starves the model of data.
This is why we have chosen to heavily focus on performance for the upcoming version 1 of tokenizers. Tokenization should be light and should scale with your workflow. Your GPUs should never sit idle waiting for the CPU to complete its tokenization.
In this article, we look at what makes v1 faster than v0.23, often by tens of times.
This work was entirely possible thanks to the rest of the ecosystem. Tokenization is a very active area of open source work, and libraries such as gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper and ai-tokenizer, as well as many others, have each pushed on what a fast tokenizer can be. We read that work, and several of the ideas below reached us because another project showed they were worth trying. Before this refactor, tokenizers was nowhere near the performance it could have had, so contributing to it may not have seemed worth it. With this refactor, we hope to make clear that we intend tokenizers to be a library worth contributing to.
We also thank NVIDIA, IBM and the ExecuTorch team for contributing patches and helping us test across a wide range of hardware to broaden platform support.
We showcase results for the release candidate of tokenizers v1 against other widely used alternatives. We go over single-threaded, multi-threaded, scaling across threads, per-model comparison, per-language comparison, latency, decoding throughput, memory heap, as well as crate size.
We run this from the tokbench repository, and add a command to rerun the benchmarks on your hardware if you would like to do so.
Native threads share one tokenizer and let the library distribute a batch across its own worker pool. Independent instances run one tokenizer per worker with no shared state. Tokenizers v1 performs best with native threads, while gigatoken performs best with independent instances.
Batch encoding matters most in data pipelines, where a tokenizer must process many inputs at once. How well it scales across CPU cores depends on how much state the threads have to share. In the M4 native-thread sweep, v1 scales from one to eight workers at 76% of linear.
Absolute throughput as workers are added, v1 against the released library, on a linear axis from zero.
Throughput determines how much text a system can process, while single-encode latency determines how long an individual inference request waits for tokenization.
The time to encode one 512-byte English document with a warm tokenizer, timed call by call. p99 is the slowest 1% of calls.
The reverse path turns token IDs back into text. Across the 6 model families measured for decode, v1 decodes at 5.4 to 8.8 times the throughput of tokenizers 0.23 on an M4 Max.
Decoded UTF-8 output in MB/s.
Runtime memory was measured with the gpt-oss tokenizer and 1.024 MB of English text on an Apple M4 Max. The selector separates one worker, one tokenizer using eight native threads, and eight independent tokenizer instances.
Live heap after loading the tokenizer and after a warm encode with the returned token buffers released.
Before v1, the Rust implementation exposed encoding, configuration loading, compatibility and training through one crate. V1 divides that implementation into smaller crates. tk-encode is the required runtime, and applications that depend directly on the subcrates can add tk-serialize, tk-convert and tk-train according to their needs.
UTF-8 uses more bytes for many characters outside Latin scripts, and tokenizer implementations can compound that cost with extra splitting work. Performance work often centers English. V1 treats non-Latin languages as part of the performance target.
The teams at Crusoe and NVIDIA make this case especially well in Reducing TTFT by CPUMaxxing Tokenization. Every prompt must be tokenized before inference can return its first token. That cost becomes visible in time to first token for long agent contexts and requests that reuse a cached model prefix.
Median p99 across eight model families, measured call by call on an Apple M4 Max. Lower is better.
The release covers more than BPE. WordPiece gained a double-array trie, a zero-allocation encode path and the shared word cache. Unigram uses the same allocation-conscious pipeline and word cache.
The gains are smaller than for BPE. These two families are where we focus next.
BPE: aggregate of headline models; WordPiece uses BERT, Unigram uses T5.
v1 will produce the same token IDs as v0.23. The goal was to preserve the output, the API, the vocabulary and the merge ranks, and improve everything that can be improved. That includes breadth. The library stays general across tokenizer families rather than specialising on BPE, so v1 loads everything v0.23 loaded.
A tokenizer converts text into the list of integers a model reads. tokenizers runs that conversion in four stages. Normalization applies operations such as lowercasing or Unicode normalization to the raw text. Pre-tokenization splits the text into smaller pieces called pre-tokens. The model turns each pre-token into tokens and maps them to IDs in its vocabulary. Post-processing adds any special tokens the model expects.
The model stage is where most of the work described here happens. Eight of the ten model families measured in this article use byte pair encoding, or BPE. BPE starts from the bytes of a pre-token and repeatedly joins the highest ranked adjacent pair until no ranked pair remains. The ranking is learned when the tokenizer is trained and ships with it, so the same text always produces the same IDs. A merge never crosses a pre-token boundary. The other two families use WordPiece and Unigram, the two other model types the library supports.
The tokenization pipeline page documents the four stages. Tokenization algorithms documents BPE, WordPiece and Unigram.
Each stage was worked on. These are the changes that mattered:
applies to most BPE tokenizers
BPE models use a regular expression to split the input text into smaller, easier to process chunks called pre-tokens. Merges happen inside a pre-token and never across the boundary between two of them, so this split decides what the rest of the pipeline sees.
That regular expression is a fixed parameter of the model. It ships with the tokenizer and never changes at runtime, so there is no need for a general-purpose regex engine to interpret it on every encode. An equivalent splitting function can be written by hand, once, for the pattern a given model actually uses.
A hand-written function can then use the SIMD instructions (single instruction, multiple data) of a modern CPU, which apply one operation to many bytes at once and suit UTF-8 text well. bitcannon views the input's bytes as parallel streams of bits, so boundaries fall out of boolean operations across whole registers instead of a scan that advances one character at a time. It decides 64 bytes per register operation. The same idea drives Parabix for text processing and simdjson for JSON.
This depends on recognising the pattern. A handful of grammars cover most byte-level BPE models, and a tokenizer whose pattern is not among them keeps the regex path and none of this speed-up. That is why the gains in section 01 vary as much as they do.
The animation illustrates how their work is structured and does not represent timings. Each step advances the regex by one byte and bitcannon by a full register. End-to-end token IDs are verified in section 01. The current public pipeline API does not expose a comparable isolated split timer for both versions, so this section does not assign a speed-up to this stage alone.
applies to BPE, WordPiece and Unigram
Real text contains many repeated words. Because BPE always produces the same token IDs for a given pre-token, v1 can save the result after processing it once. A thread-local cache maps each pre-token's bytes to its token IDs, allowing later occurrences to skip the merge process.
Naturally, as the input grows, the number of unique words can grow more slowly than the total number of words. Repeated words then account for an increasing share of the input. New words still appear, which accounts for the occasional misses in the animation below.
The bar represents the work required to convert each pre-token into token IDs. A cache miss runs the full BPE merge loop, so the bar fills slowly. A cache hit requires only a lookup and finishes sooner. The animation is schematic.
Cache enabled divided by cache disabled, measured on an Apple M4 Max. Corpus bars are medians across 8 model families in one complete report. Shared prompt prefix is the median across 3 reports and 8 model cells per report. Each report uses 100 distinct 10 KiB requests from a real agent trace, with a shared 8 KiB prefix and a different suffix. Both configurations produced the same token IDs.
Reproduce the shared-prefix result with tokbench measure prefix-sharing --engine pipeline --engine hf-tokenizers --compare-to pipeline-no-cache --corpus agentic_swe.
applies to BPE only
The next major cost comes from the BPE merge loop. For each pre-token, the loop repeatedly finds the highest-priority adjacent pair and merges it. The previous implementation allocated new memory for every call and built a new priority queue for every pre-token.
v1 reuses a scratch buffer owned by the caller, removing those repeated allocations. It stores symbols in a flat array and links adjacent symbols by their positions in that array, which makes updates during merging cheaper. It also processes a batch of pre-tokens in a single model call.
Each candidate pair is also packed into a single 64-bit value, with the merge rank in the high bits. Comparing two candidates is then just comparing two integers, and "no merge here" is the largest possible value, so the loop finds its next merge without a branch.
Small differences in benchmark design can produce large differences in tokenizer performance. We used the following rules to keep the comparison consistent across engines.
Both conditions are sometimes described as "warm," even though they measure different workloads. Our headline results use distinct documents, and the complete corpus is too large to fit in the cache. Tokenizer benchmarks should identify which workload they use because the choice can dominate the result.
Across the ten model families v1's encode path covers, it encodes text 3 to 30 times faster than v0.23 with one thread on an Apple M4 Max. The low end is t5-base, the high end gpt2. It scales at 76% of linear across eight workers. Throughout these changes, v1 produces exactly the same token IDs as the released library.
The overall improvement comes from several changes working together: a hand-written splitter in place of a regex engine, a cache that answers a repeated word without merging it again, a merge loop that never touches the allocator, and one model call per batch of pre-tokens instead of one per pre-token. Each reduces the work done at a different point in the pipeline.
The next priority is support for more model families. We will move additional models onto the new merge loop before 1.0.0. The following section tracks that work.
This page is generated from tokbench results and will be updated as support expands.
A release candidate for v1 is on crates.io. The API you call is the one you already call, so the only thing that changes is which build you install.
It is the ordinary install:
bashcargo add tokenizers --pre
Training is behind a default-on feature that pulls a C++ dependency with it. If you only need to encode, turn it off to exclude the training implementation:
bashcargo add tokenizers --pre --no-default-features --features http
Encoding is unchanged: same call, same ids.
rustuse tokenizers::tokenizer::{Result, Tokenizer}; fn main() -> Result<()> { let tokenizer = Tokenizer::from_pretrained("deepseek-ai/DeepSeek-V4-Flash", None)?; let encoding = tokenizer.encode("The tokenizer is no longer the bottleneck.", false)?; println!("{:?}", encoding.get_ids()); // [671, 17840, 9160, 344, 1119, 5827, 270, 111127, 16] println!("{:?}", encoding.get_tokens()); // ["The", "Ġtoken", "izer", "Ġis", "Ġno", "Ġlonger", "Ġthe", "Ġbottleneck", "."] Ok(()) }
For a batch, encode_batch is what scales across cores. It is the call the threads section measures.
rustlet encodings = tokenizer.encode_batch(documents, false)?;
Every figure on this page was measured against this crate. The Python bindings wrap the same code and are built from bindings/python, but they add per-call overhead that none of these measurements include.
The benchmarks on this page cover the completed release-candidate work listed first. The remaining sections show what is still required for 1.0.0 and what we plan to explore afterward.
█ shipped ▓ in progress · planned
This work is in the Rust pre-release on crates.io. Install it with cargo add tokenizers --pre.