String Tokenizer for Node.js using ICU's BreakIterators
String Tokenizer for Node.js using ICU's BreakIterators
No description provided.
install 'full-icu' data for your current node
Tokenize CSS
A robust Punycode converter that fully complies to RFC 3492 and RFC 5891, and works on nearly all JavaScript platforms.
A promise based streaming tokenizer
Tokenized zip support
ICU message format compiler with a <1KB runtime bundle footprint
TypeScript definition for strtok3 token
A robust Punycode converter that fully complies to RFC 3492 and RFC 5891, and works on nearly all JavaScript platforms.
No description provided.
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
convert hanzi to pinyin
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
i18nFormat plugin to use ICU format with i18next
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parse CSS media query lists.
ProseMirror Markdown integration
Shows code error fragment of input file
tokenizer of source code for jscpd