Small library that provides functions to tokenize a string into an array of words with or without punctuation
r/w stream of glsl tokens
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
safely inject a block of tokens into a shader
Solve CSS math expressions
Trim the whitespace within an array of GLSL tokens
Infer the scope of each token in an array of GLSL tokens
Determine the scope depth of an array of GLSL tokens
"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.
A promise based streaming tokenizer
Tokenize CSS
Tokenized zip support
TypeScript definition for strtok3 token
Retrieve the values defined with preprocessor statements in a selection of GLSL tokens
Algorithms to help you parse CSS from an array of tokens.
Converts an array of GLSL tokens to a plain source string
Parses and stringifies CSS selectors
A tokenzier for Sass' SCSS syntax
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
core functionality of copy/paste detector for jscpd
Extract function definitions from an array of GLSL tokens.
Common token types for decoding and encoding numeric and string values
A simple string tokenizer designed to capture punctuation and sentence flow information.
RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant as a lexer for Ruby that's complete and correct; all legal Ruby code should be lexed correctly by RubyLexer as well. Just enough parsing capability is included to give RubyLexer enough context to tokenize correctly in all cases. (This turned out to be more parsing than I had thought or wanted to take on at first.) RubyLexer handles the hard things like complicated strings, the ambiguous nature of some punctuation characters and keywords in ruby, and distinguishing methods and local variables. It should be able to correctly lex 99.9+% of legal ruby code.