JS/WASM bindings for tiktoken
JavaScript port of tiktoken
JS/WASM bindings for tiktoken
_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
This is the **x86_64-unknown-linux-gnu** binary for `tiktoken-rs-node`
This is the **x86_64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
JavaScript port of tiktoken
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Tiktoken but thicker; A bundled tokenizer with added helper functions
This is the **aarch64-apple-darwin** binary for `tiktoken-rs-node`
CLI for counting tokens in files and directories using tiktoken
JavaScript port of tiktoken
WASM bindings for the tiktoken BPE tokenizer
This is the **x86_64-apple-darwin** binary for `@anysphere/tiktoken-node`
This is the **aarch64-apple-darwin** binary for `@anysphere/tiktoken-node`
This is the **x86_64-pc-windows-msvc** binary for `@anysphere/tiktoken-node`
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
This is the **aarch64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
Tiktoken tokenizer integration for harness-one
This is the **aarch64-unknown-linux-musl** binary for `tiktoken-rs-node`
This is the **aarch64-unknown-linux-gnu** binary for `tiktoken-rs-node`
JavaScript port of tiktoken
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken
Library for encoding and decoding with the tiktoken library in Rust
Library for encoding and decoding with the tiktoken library in Rust
Fast byte-pair encoding implementation.
entelix vendor-accurate TokenCounter for OpenAI BPE encodings (cl100k_base, o200k_base, p50k_base, r50k_base) — wraps tiktoken-rs with eager preload
Streaming token counter for partial LLM responses. Accumulates token count across chunks without holding the full text. Pluggable estimator function. Zero deps.
WASM bindings for the tiktoken BPE tokenizer
a fast BPE tokeniser for use with OpenAI's models
tiktoken (OpenAI BPE) library support for toktrie and llguidance
LLM tokenizer library with caching and chat template support
A BPE (Byte Pair Encoding) tokenizer written in Rust with Python bindings
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs.
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.
Tiktoken tokenizer for Roseflow.
A pure Ruby implementation of OpenAI's tiktoken library for BPE tokenization
JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base.
[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.