A simple JavaScript implementation of Byte Pair Encoding (BPE)
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
Tokenizer for OpenAI large language models.
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
TypeScript definition for strtok3 token
[](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [](https://ww
Algorithms to help you parse CSS from an array of tokens.
JS tokenizer for LLaMA-based LLMs
Parses and stringifies CSS selectors
A tokenzier for Sass' SCSS syntax
Solve CSS math expressions
Multi-arch builds of HuggingFace tokenizers
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
JS tokenizer for LLaMA 3
Common token types for decoding and encoding numeric and string values
Sniff the encoding from a HTML byte stream
tokenizer of source code for jscpd
Implementation of the HTML encoding sniffer algo, with stream support
Convert encodings, uses iconv-lite
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.