BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us

llm-tokenizer

v1.3.2crates.io· Rust

LLM tokenizer library with caching and chat template support

The verdict
Worth a look. Actively maintained and growing, actively maintained.
Live from the crates.io registry · derived rules, not AI
How it scores
MaintenanceHealthy
PopularityRising
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 1 month ago.
Popularity
39K downloads / week
Security
No known advisories for this version (OSV).
License
Apache-2.0
Dependencies
No runtime dependencies
Recent releases
  • 1.3.21 month ago
  • 1.3.11 month ago
  • 1.3.03 months ago
  • 1.2.03 months ago
  • 1.1.04 months ago
  • 1.0.04 months ago
  • 0.0.15 months ago
llm-tokenizer — LLM tokenizer library with caching and chat template support (Rust / crates.io) — verdict, alternatives & security · Modules