Open Source
tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Our verdict
We have not reviewed tokenizers yet
No one at Enterprise DNA has checked this listing against its sources. The description comes from public listings gathered automatically. Any fact below shows where it came from and when it was last checked.
From the listing
About tokenizers
A Rust implementation of fast tokenizers, optimized for both research and production NLP pipelines. It provides subword tokenization algorithms such as BPE, WordPiece, and Unigram with full alignment tracking. The library is framework-agnostic and includes Python bindings for easy integration.
Best for
Developers needing high-throughput tokenization for NLP model training or serving
Use cases
- Tokenizing large text corpora for model training
- Integrating tokenization into production inference systems
- Building custom tokenizers for specialized vocabularies
Notes
10,782 stars on GitHub. Last updated 2026-05-26. Licensed Apache-2.0.
Pros
- Blazingly fast performance due to Rust implementation
- Supports multiple tokenization algorithms with consistent API
- Seamless Python bindings for integration with ML workflows
Cons
- Limited to tokenization tasks without broader NLP utilities
- Requires compilation for Rust or using pre-built wheels
- Smaller community compared to Python-native tokenizers
Key facts
What we have checked
- Latest prerelease version
- 1.0.0-rc.4 (released 9 Nov 2026)
- Days since the last commit
- 0
- GitHub forks
- 1,257
- GitHub stars
- 11,260
- Days to a first reply on issues
- 23.2
- Open issues
- 82
- Releases in the last 90 days
- 5
Each value shows when we last confirmed it and where it came from. Values we have not confirmed are left out, never guessed.
Stars and usage
How tokenizers is growing
10,782 to 11,260
Health and security
Is tokenizers looked after, and what do its setup files show
Health score
81 out of 100
| Part | Points | What we saw |
|---|---|---|
| Recent activity | 30 of 30 | Last push 0 days ago |
| Releases | 15 of 15 | 5 releases in 90 days |
| Issue replies | 5 of 20 | First reply in 23.2 days (median) |
| Maintainer | 15 of 15 | Owned by an organisation |
| Official source | Not counted | MCP servers only |
Sources
Where this comes from
A source shows what was published when we checked it. It does not certify security, performance or a working installation. Read our method.
Changelog
What changed
Released 1.0.0-rc.4
Released 1.0.0-rc.3
Released 0.23.3
Released 1.0.0-rc.2
Released 0.23.2
Released 0.23.1
Released 0.22.2
Released 0.22.1
Released 0.22.0
Released 0.21.4
Released 0.21.3
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
Jax
Community
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
PyTorch
Community
Tensors and Dynamic neural networks in Python with strong GPU acceleration
TensorFlow
Community
An Open Source Machine Learning Framework for Everyone
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.