Convert SQL statements into a list of tokens
SQL tokenizer and parser for AxiomDB — shared by babel-plugin and vscode-extension
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
A tokenzier for Sass' SCSS syntax
Solve CSS math expressions
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
Utilities for SQL instrumentations
r/w stream of glsl tokens
Claude tokenizer
Tokenizes a string that represents a regular expression.
Replacement for sql-tokenizer which supports Mysql syntax. Also provides an experimental utility for structuring tokens into SQL statements.
Format whitespace in a SQL query to make it more readable
Tokenize a shell string into argv array
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Parse CSS media query lists.
Presto SQL Parser for Ruby parses a SQL using Presto's native SQL parser precisely and reports error if given SQL has syntax errors. Optionally, it returns a ANTLR tokens sequence.
Trino SQL Parser for Ruby parses a SQL using Trino's native SQL parser precisely and reports error if given SQL has syntax errors. Optionally, it returns a ANTLR tokens sequence.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.