Simple string tokenizer, to make RegExps less painful.
Solve CSS math expressions
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
r/w stream of glsl tokens
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
core functionality of copy/paste detector for jscpd
safely inject a block of tokens into a shader
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
Trim the whitespace within an array of GLSL tokens
"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.
Infer the scope of each token in an array of GLSL tokens
Determine the scope depth of an array of GLSL tokens
Tokenizer for OpenAI large language models.
My JavaScript parser