Efficiently modify strings containing ANSI escape codes
Transform stream that tokenizes CSS
Tokenize a string into an array of string parts and format identifier objects.
A tokenzier for Sass' SCSS syntax
transform stream to tokenize html
Crypto icons by Ledger
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
No description provided.
Tokenize a string.
Parse CSS color values
Tokenize CSS
No description provided.
No description provided.
No description provided.
Provide a high level wrapper for kuromoji.js
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
Tokenize Excel formulas
The lexer for Materialize's SQL dialect, with wasm build targets.
Small library that provides functions to tokenize a string into an array of words with or without punctuation
tokenize a string that includes ansi code
Multilingual tokenizer that automatically tags each token with its type
micromark utility to tokenize subtokens
match a tokenized html stream with css selectors
Estimate the number of tokens for Gemini models