tokenizers

v0.23.1crates.io· Rust

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

The verdict

Solid choice. Well-adopted and active, actively maintained.

Live from the crates.io registry · derived rules, not AI

How it scores

MaintenanceHealthy

PopularityWidely used

SecurityClean

LicensePermissive

DepsZero deps

Maintenance

Last published 1 month ago.

Popularity

499K downloads / week

Security

No known advisories for this version (OSV).

License

Apache-2.0

Dependencies

No runtime dependencies

Recent releases