TS tokenizer for Mistral-based LLMs
TS tokenizer for Mistral-based LLMs
TS tokenizer for Mistral-based LLMs
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
tokenizer of source code for jscpd
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parse CSS media query lists.
ProseMirror Markdown integration
Parse CSS Cascade Layer names.
Common token types for decoding and encoding numeric and string values
detector of copy/paste in files
gemma3 tokenizer for NodeJS/Browser
Parse CSS color values
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
core functionality of copy/paste detector for jscpd
Tokenizer for OpenAI large language models.