discourse_ai-tokenizers
v0.4.2RubyGems· RubyA Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
The verdict
Worth a look. Actively maintained and growing, actively maintained.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceHealthy
PopularityRising
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 3 months ago.
Popularity
11K downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
- 0.4.23 months ago
- 0.4.13 months ago
- 0.45 months ago
- 0.3.26 months ago
- 0.3.111 months ago
- 0.3.011 months ago
- 0.2.011 months ago
- 0.1.211 months ago