BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us
modules.com
⌘K

rz_tiktoken_ruby

v0.0.6RubyGems· Ruby

An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.

The verdict
Abandoned. Last published 2 years ago. No recent activity — look for a maintained alternative.
No recent activity — look for a maintained alternative.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceAbandoned
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 2 years ago.
Popularity
16 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
  • 0.0.62 years ago
rz_tiktoken_ruby — An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it. (Ruby / RubyGems) · Modules