A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
Tokenized zip support
Tokenize CSS
A promise based streaming tokenizer
TypeScript definition for strtok3 token
LLM Tokenizer - Count tokens for LLM prompts and responses. | 大模型分词器 - 统计大模型提示词与回复的 token 数量。
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
A tokenzier for Sass' SCSS syntax
Solve CSS math expressions
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Tokenizer for OpenAI large language models.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
r/w stream of glsl tokens
Claude tokenizer
Tokenizes a string that represents a regular expression.
Parse CSS media query lists.
detector of copy/paste in files
Tokenize a shell string into argv array
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
TOON is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage.
LLM Token Cost Monitor
A Minitest reporter optimized for Large Language Model consumption, featuring compact emoji-based output, regression detection by comparing test runs, TOML report generation, and detailed failure reporting with file locations. Perfect for AI-assisted development workflows.
llm_optimizer reduces LLM API costs by up to 80% through semantic caching, intelligent model routing, token pruning, and conversation history summarization. Strictly opt-in and non-invasive.
LLM Rescuer uses artificial intelligence to guess what you probably meant when you called a method on nil. Instead of crashing with NoMethodError, it asks GPT to analyze your code and hallucinate a reasonable response. Because clearly, the best way to solve Tony Hoare's billion-dollar mistake is to throw AI at it until it works. What could possibly go wrong? 🎭
A simple Rails engine to log API usage from multiple LLM providers and provide methods for tracking user consumption over time, enabling easy rate-limiting.
Captures LLM token usage metrics per task for cost attribution and intelligent routing
Compact serialization format optimized for LLM contexts with 30-60% token reduction compared to JSON
CTON provides a JSON-compatible, token-efficient text representation optimized for LLM prompts.
Lightweight Ruby library for converting JSON data to TOON format, achieving 30-60% token reduction for LLM applications
A Ruby MCP server that wraps Linear's GraphQL API and returns TOON-formatted responses for ~40-60% token savings in LLM workflows.
A Ruby gem for converting between JSON and TOON format. TOON is a compact serialization format designed for LLMs that reduces token usage by 30-60% compared to JSON. Supports bidirectional conversion, tabular arrays, nested structures, and lossless roundtrips.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.