gemini tokenizer for NodeJS/Browser
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
TypeScript definition for strtok3 token
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Algorithms to help you parse CSS from an array of tokens.
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
Solve CSS math expressions
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
Google AI JavaScript SDK
Parse CSS media query lists.
detector of copy/paste in files
[](https://www.npmjs.com/package/@google/genai) [](https://www.npmjs.com/package/@google/genai)
Tokenize a shell string into argv array
Tiny JavaScript tokenizer.
Authoritative Gemini tokenizer for Rust, ported from the official Google Python GenAI SDK
Terminal UI to chat with large language models (LLM) using different model backends, and direct integrations with your favourite editors!
Count tokens for LLM models using exact tokenization
A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
AgentHarness provides a unified interface for CLI-based AI coding agents like Claude Code, Cursor, Gemini CLI, and others. It offers full orchestration with provider switching, circuit breakers, health monitoring, flexible configuration, dynamic provider registration, and token usage tracking.
LLM Conductor provides a clean, unified interface for working with multiple Language Model providers including OpenAI GPT, Anthropic Claude, Google Gemini, Groq, OpenRouter, and Ollama. Features include prompt templating, token counting, and extensible client architecture.
Logs every call your Rails app makes to OpenAI, Anthropic, Gemini, RubyLLM, or an OpenAI-compatible API: tokens, cost, latency, tags. Calls go straight to the provider — no proxy. Includes price sync, budget guardrails, and a mountable dashboard.