RWKV / gpt-NeoX / Pythia, 0-dep tokenizer library, for nodejs
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
A tokenzier for Sass' SCSS syntax
Solve CSS math expressions
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parse CSS media query lists.
ProseMirror Markdown integration
Shows code error fragment of input file
Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
<!-- :begin use tokenizer/banner -->
Claude tokenizer
r/w stream of glsl tokens
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Tokenizer for processing inline math (formulas)