A regular expression (RexExp) stream tokenizer.
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Tokenized zip support
Tokenize CSS
A promise based streaming tokenizer
Escape RegExp special characters
TypeScript definition for strtok3 token
Express style path to RegExp utility
r/w stream of glsl tokens
My JavaScript parser
Algorithms to help you parse CSS from an array of tokens.
split a Text Stream into a Line Stream, using Stream 3
Compile ESNext Regular Expressions to ES5
Compile inline regular expression modifiers
Parses and stringifies CSS selectors
Regular Expressions parser in JavaScript
Solve CSS math expressions
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
A tokenzier for Sass' SCSS syntax
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
A streaming data transport format that aims to support built-in features such as Promises, Dates, RegExps, Maps, Sets and more.
ProseMirror Markdown integration
Check if a value is a regular expression
Like JSONStream, but using Minipass streams