Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Tokenized zip support
Tokenize CSS
A promise based streaming tokenizer
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
tokenizer of source code for jscpd
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
My JavaScript parser
ProseMirror Markdown integration
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Tokenize a shell string into argv array
Common token types for decoding and encoding numeric and string values
JS tokenizer for LLaMA-based LLMs
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
Parse CSS media query lists.
A simple HTML lexer/tokenizer written in Ruby.
Contentful API wrapper library exposing an ActiveRecord-like interface