A fast tokenizer/lexer for JavaScript
A fast tokenizer/lexer core that is well tested.
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
moo (ultra-fast tokenizer) plugin for nearley
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
tokenizer of source code for jscpd
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Fast tokenizer.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Common token types for decoding and encoding numeric and string values
High-performance library for tokenizing text.
r/w stream of glsl tokens
Claude tokenizer
Fast state-of-the-art tokenizers for Ruby
Fast string tokenizer. Nom strings.
JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base.
[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.
Fast B-tree–backed token store for stateful user sessions. Provides authentication and authorization across multiple processes. Optimized for vertical scaling on a single server
TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.
A Ruby gem to verify the signature of Firebase ID Tokens. It uses Redis to store Google's x509 certificates and manage their expiration time, so you don't need to request Google's API in every execution and can access it as fast as reading from memory.
Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.