A Simple Byte-Pair Encoding (BPE) tokenizer built from scratch.
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
TypeScript definition for strtok3 token
Tokenizer for OpenAI large language models.
JS tokenizer for LLaMA-based LLMs
Algorithms to help you parse CSS from an array of tokens.
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
r/w stream of glsl tokens
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
JS tokenizer for LLaMA 3
Tiny JavaScript tokenizer.
Common token types for decoding and encoding numeric and string values
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
tokenizer of source code for jscpd
Multi-arch builds of HuggingFace tokenizers
Tokenizes a string that represents a regular expression.
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
[](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [](https://ww
ProseMirror Markdown integration