Results for regex-tokenizer

Tokenize a string into multiple token classes using a set of regular expressions, and get them back in order!

Minimal Regex Tokenizer.

general purpose regex tokenizer that returns an array of tokens

Tiny JavaScript tokenizer.

Tokenize CSS

A promise based streaming tokenizer

Tokenized zip support

A regular expression to match all Emoji-only symbols as per the Unicode Standard.

TypeScript definition for strtok3 token

Give a regex, get a robust predicate function that tests it against a string.

Algorithms to help you parse CSS from an array of tokens.

Is this value a JS regex? Works cross-realm/iframe, and despite ES6 @@toStringTag

Pass two numbers, get a regex-compatible source string for matching ranges. Validated against more than 2.78 million test assertions.

Compile regular expressions using named groups to ES5.

Compile ES2015 Unicode regex to ES5

Parses and stringifies CSS selectors

Compile ES2015 sticky regex to an ES5 RegExp constructor

Regular expression for matching ANSI escape codes

Regular expression for matching a shebang line

Compile Unicode property escapes in Unicode regular expressions to ES5.

Compile regular expressions using the `s` (`dotAll`) flag to ES5.

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

Solve CSS math expressions

Compile regular expressions' unicodeSets (v) flag.