A promise based streaming tokenizer
Tokenize CSS
Tokenized zip support
Features from the rust language in javascript: Provides Traits/Type classes & an advanced library for working with sequences/iterators in js.
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
Solve CSS math expressions
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Common token types for decoding and encoding numeric and string values
tokenizer of source code for jscpd
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
Claude tokenizer
Tokenize a shell string into argv array
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Parse CSS media query lists.
detector of copy/paste in files
Tiny JavaScript tokenizer.
core functionality of copy/paste detector for jscpd
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
Tokenization wrapper for Ferrum inference engine
Custom CUDA kernels and decode runner for Ferrum inference
Backend implementations (Candle, CPU) for Ferrum inference
CLI for Ferrum — a Rust-native LLM inference engine
Model orchestration engine for Ferrum LLM inference
Core trait contracts for the Ferrum LLM inference engine
Unified compute kernels (CUDA/Metal/CPU) and model runner for Ferrum inference
KV cache management with PagedAttention for Ferrum inference
Model architectures (LLaMA, Qwen, BERT) for Ferrum inference
Weight-format abstraction (Dense / GPTQ / AWQ / GGUF) for Ferrum models
Sampling strategies for Ferrum LLM inference engine
Request scheduling for Ferrum LLM inference engine