Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
r/w stream of glsl tokens
Claude tokenizer
Tokenizes a string that represents a regular expression.
Parse CSS media query lists.
detector of copy/paste in files
Tokenize a shell string into argv array
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Multilingual tokenizer that automatically tags each token with its type
Tiny JavaScript tokenizer.
gemma3 tokenizer for NodeJS/Browser
Tokenizer for OpenAI large language models.
Pure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby` that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).
A pure-Ruby, zero-dependency implementation of the Vercel AI SDK "Data Stream Protocol" (UI Message Stream Protocol) — the Server-Sent-Events wire format that drives the AI SDK's useChat / useCompletion / useObject frontend hooks. The protocol is language-agnostic by design, but Ruby had no implementation; ai_stream lets a Rails/Rack backend stream text, reasoning, tool calls, sources, files, and custom data parts to a Vercel-AI-SDK frontend with the exact frames it expects. Provider-agnostic: it composes with ruby_llm, ruby-openai, or any token source instead of competing with them.