Tokenizes a string that represents a regular expression.
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
JavaScript implementation of Japanese morphological analyzer
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Parse CSS media query lists.
Multilingual tokenizer that automatically tags each token with its type
detector of copy/paste in files
gemma3 tokenizer for NodeJS/Browser
Tokenizer for OpenAI large language models.
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
Parse CSS Cascade Layer names.
Canvas graphics API backed by Cairo
Shows code error fragment of input file
Super-fast alternative to Babel for when you can target modern JS runtimes
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
Parse CSS color values
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.
Helpers for creating and serializing transactions
A primitive to tokenize your solid-components to enable custom parsing.
JS tokenizer for LLaMA 3
A Node.js gRPC library that is nice to you
<!-- :begin use tokenizer/banner -->
CSS / Extended CSS tokenizer