tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_
This is the **x86_64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
This is the **x86_64-apple-darwin** binary for `@anysphere/tiktoken-node`
This is the **aarch64-apple-darwin** binary for `@anysphere/tiktoken-node`
This is the **x86_64-pc-windows-msvc** binary for `@anysphere/tiktoken-node`
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
This is the **aarch64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
This is the **aarch64-pc-windows-msvc** binary for `@anysphere/tiktoken-node`
JavaScript port of tiktoken
JS/WASM bindings for tiktoken
JS/WASM bindings for tiktoken
This is the **x86_64-unknown-linux-gnu** binary for `tiktoken-rs-node`
This is the **aarch64-apple-darwin** binary for `tiktoken-rs-node`
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
JS tokenizer for LLaMA-based LLMs
Tokenizer for OpenAI large language models.
JavaScript port of tiktoken
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Tiktoken but thicker; A bundled tokenizer with added helper functions
Estimate the number of tokens an OpenAI chat completion request will use
Node.js Streams, a user-land copy of the stream library from Node.js