Open Source
tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Our verdict
We have not reviewed tokenizers yet
No one at Enterprise DNA has checked this listing against its sources. The description comes from public listings gathered automatically. Any fact below shows where it came from and when it was last checked.
From the listing
About tokenizers
A Rust implementation of fast tokenizers, optimized for both research and production NLP pipelines. It provides subword tokenization algorithms such as BPE, WordPiece, and Unigram with full alignment tracking. The library is framework-agnostic and includes Python bindings for easy integration.
Best for
Developers needing high-throughput tokenization for NLP model training or serving
Use cases
- Tokenizing large text corpora for model training
- Integrating tokenization into production inference systems
- Building custom tokenizers for specialized vocabularies
Notes
10,782 stars on GitHub. Last updated 2026-05-26. Licensed Apache-2.0.
Pros
- Blazingly fast performance due to Rust implementation
- Supports multiple tokenization algorithms with consistent API
- Seamless Python bindings for integration with ML workflows
Cons
- Limited to tokenization tasks without broader NLP utilities
- Requires compilation for Rust or using pre-built wheels
- Smaller community compared to Python-native tokenizers
Key facts
What we have checked
- Latest prerelease version
- 1.0.0-rc.3 (released 19 Oct 2026)
- GitHub stars
- 11,116
Each value shows when we last confirmed it and where it came from. Values we have not confirmed are left out, never guessed.
Stars and usage
How tokenizers is growing
10,782 to 11,116
Sources
Where this comes from
- GitHubOfficial
- github api via nightly crawl, recovered from git history 20e874270887c1008f860ee0198e461a23c9e438Registry
A source shows what was published when we checked it. It does not certify security, performance or a working installation. Read our method.
Changelog
What changed
Released 1.0.0-rc.3
Released 0.23.3
Released 1.0.0-rc.2
Released 0.23.2
Released 0.23.1
Released 0.22.2
Released 0.22.1
Released 0.22.0
Released 0.21.4
Released 0.21.3
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
Jax
Community
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
PyTorch
Community
Tensors and Dynamic neural networks in Python with strong GPU acceleration
TensorFlow
Community
An Open Source Machine Learning Framework for Everyone
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.