JavaScript port of tiktoken
JavaScript port of tiktoken
JS/WASM bindings for tiktoken
JS/WASM bindings for tiktoken
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
JavaScript port of tiktoken
Browser-compatible token counter for LLM content blocks using js-tiktoken
_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_
This is the **x86_64-unknown-linux-gnu** binary for `tiktoken-rs-node`
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
This is the **x86_64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
Browser-compatible token counter for LLM content blocks using js-tiktoken
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
JavaScript port of tiktoken
JS tokenizer for LLaMA-based LLMs
This is the **aarch64-apple-darwin** binary for `tiktoken-rs-node`
Estimate the number of tokens an OpenAI chat completion request will use
Tokenizer for OpenAI large language models.
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Tiktoken but thicker; A bundled tokenizer with added helper functions
Tokenizer utilities for Tokenlens.
JavaScript/TypeScript Implementation of LLMLingua-2
JavaScript port of tiktoken that works in MongoDB
JavaScript port of tiktoken