React component to convert a text string into React.ReactNodes
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
ProseMirror Markdown integration
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
TypeScript definition for strtok3 token
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Claude tokenizer
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
<!-- :begin use tokenizer/banner -->
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
r/w stream of glsl tokens
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Tokenizes a string that represents a regular expression.
Parse CSS media query lists.
detector of copy/paste in files
Tokenize a shell string into argv array