tok3niz3r-core
v0.2.0crates.io· RustCore types and on-disk schema for the tok3niz3r byte-level BPE tokenizer: token ids, vocabulary, merge table, and the GPT-2 byte-level mapping.
The verdict
Maintained. Niche but maintained, actively maintained.
Live from the crates.io registry · derived rules, not AI
How it scores
MaintenanceHealthy
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published this month.
Popularity
3 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
- 0.2.0this month
- 0.1.0this month