The API for building streaming tokenizers and lexers.
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
Solve CSS math expressions
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
ContractCase HTTP plugin CaseExample definition DSL
ContractCase core function plugin DSL
Tokenize a shell string into argv array
Types for writing the definition DSL for plugins for the ContractCase test framework
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Parse CSS media query lists.
Definition DSL components for the ContractCase contract testing suite
Create beautiful Javascript documentations with this ruby-gem. It's pretty easy to customize and add your own tokens/DSL.
Named browser sessions, Ruby workflow DSL, and a token-efficient DOM snapshot format. Built on a browser-agnostic driver layer (Ferrum/CDP backend).
It wraps any LLM client with a tiny DSL. In test environement, it records “tapes” (YAML fixtures of real LLM calls) and replays them on subsequent runs; when a tape is stale, it re-records to keep tests current. Production stays clean and safe, while CI avoids hammering the API every run--yielding deterministic tests, faster pipelines, and fewer tokens spent.