JavaScript port of tiktoken
JS/WASM bindings for tiktoken
JS/WASM bindings for tiktoken
_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
This is the **x86_64-unknown-linux-gnu** binary for `tiktoken-rs-node`
Tokenizer for OpenAI large language models.
No description provided.
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Rust language support for the CodeMirror code editor
This is the **x86_64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`
Rust dictionary for cspell.
Super-fast alternative for babel
Lezer-based Rust grammar
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
Welcome to the [Node.js] binding for the Rust [`matrix-sdk-crypto`] library! This binding is part of the [`matrix-rust-sdk`] project, which is a library implementation of a [Matrix] client-server.
Rust grammar for tree-sitter
Generates and consumes source maps
Queue for messages and jobs based on Redis
Browser automation CLI for AI agents
This is the **aarch64-apple-darwin** binary for `tiktoken-rs-node`
A high-performance utilities for WhatsApp, powered by Rust and WebAssembly.
Create cross-toolchain from https://github.com/rust-cross/manylinux-cross/tree/main
A simple toolkit for building scalable and maintainable applications
JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base.
[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.