tokenizers
v0.23.1crates.io· RustProvides an implementation of today's most used tokenizers, with a focus on performances and versatility.
The verdict
Solid choice. Well-adopted and active, actively maintained.
Live from the crates.io registry · derived rules, not AI
How it scores
MaintenanceHealthy
PopularityWidely used
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 1 month ago.
Popularity
499K downloads / week
Security
No known advisories for this version (OSV).
License
Apache-2.0
Dependencies
No runtime dependencies
Recent releases
- 0.23.11 month ago
- 0.22.26 months ago
- 0.22.18 months ago
- 0.22.09 months ago
- 0.21.410 months ago
- 0.21.4-dev.011 months ago
- 0.21.211 months ago
- 0.21.1over a year ago