Compile javascript into an AST to be consumed by other tools
A promise based streaming tokenizer
Tokenize CSS
Tokenized zip support
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
TypeScript definition for strtok3 token
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Common token types for decoding and encoding numeric and string values
ProseMirror Markdown integration
Algorithms to help you parse CSS from an array of tokens.
tokenizer of source code for jscpd
Tiny JavaScript tokenizer.
Parses and stringifies CSS selectors
A tokenzier for Sass' SCSS syntax
Solve CSS math expressions
Tesseract C++ API in Pure Javascript
Tokenize a shell string into argv array
core functionality of copy/paste detector for jscpd
RE2JS is the JavaScript port of RE2, a regular expression engine that provides linear time matching
My JavaScript parser
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
Claude tokenizer
r/w stream of glsl tokens
JS tokenizer for LLaMA-based LLMs