Replacement for sql-tokenizer which supports Mysql syntax. Also provides an experimental utility for structuring tokens into SQL statements.
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
OpenTelemetry instrumentation for `mysql` database client for MySQL
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
TypeScript definitions for mysql
fast mysql driver. Implements core protocol, prepared statements, ssl and compression in native JS
r/w stream of glsl tokens
Claude tokenizer
Tokenizes a string that represents a regular expression.
AWS RDS SSL certificates bundles.
Parse CSS media query lists.
Tokenize a shell string into argv array
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.