A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
CLI to see GPT stats based on gpt-tokenizer package
gpt-tokenizer
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Tokenize CSS
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
Tokenizer for OpenAI large language models.
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
[](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [](https://ww
JS tokenizer for LLaMA-based LLMs
n8n node for working with BPE Tokens with OpenAI's GPT models in mind.
r/w stream of glsl tokens
A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.
LLM Conductor provides a clean, unified interface for working with multiple Language Model providers including OpenAI GPT, Anthropic Claude, Google Gemini, Groq, OpenRouter, and Ollama. Features include prompt templating, token counting, and extensible client architecture.
Pure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby` that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.