No description provided.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Tokenized zip support
Tokenize CSS
A promise based streaming tokenizer
tokenizer of source code for jscpd
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
ProseMirror Markdown integration
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Common token types for decoding and encoding numeric and string values
a lightweight no-dependency fork of transformers.js (only tokenizers)
JS tokenizer for LLaMA-based LLMs
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
Parse CSS media query lists.
detector of copy/paste in files
JS tokenizer for LLaMA 3
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!