ruby_llm-tokenizer
v0.1.0RubyGems· RubyPure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby` that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).
The verdict
Maintained. Niche but maintained, actively maintained.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceHealthy
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published this month.
Popularity
112 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
- 0.1.0this month