Tokenize a shell string into argv array
Tokenized zip support
Tokenize CSS
A promise based streaming tokenizer
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
Parse CSS media query lists.
detector of copy/paste in files
Tiny JavaScript tokenizer.
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
gemma3 tokenizer for NodeJS/Browser
Multilingual tokenizer that automatically tags each token with its type
Parse CSS Cascade Layer names.
Strip the first token off a string, pass remainder of string as arg to function identified by first token
Contentful API wrapper library exposing an ActiveRecord-like interface