No description provided.
JS tokenizer for LLaMA-based LLMs
JS tokenizer for LLaMA 3
JS tokenizer for Mistral-based LLMs
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
r/w stream of glsl tokens
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Tiny JavaScript tokenizer.
Common token types for decoding and encoding numeric and string values
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
tokenizer of source code for jscpd
Tokenizes a string that represents a regular expression.
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
ProseMirror Markdown integration
BPE tokenizer used for the NovelAI frontend.
Claude tokenizer
Tokenize a shell string into argv array
No description provided.
No description provided.