Convert Oniguruma patterns to native JavaScript RegExp
deepseek_v3 tokenizer for NodeJS/Browser
Fork of @lexical/code with Prism.js syntax highlighting removed. Drop-in replacement for @lexical/code.
Simple string tokenizer for lexical scanning operations
Extract function definitions from an array of GLSL tokens.
Ruby plugin for textlint
This is a no-dependency, super lite version of IP2Location LITE lookup.
llama3 tokenizer for NodeJS/Browser
qwen3 tokenizer for NodeJS/Browser
SockJS-node is a server counterpart of SockJS-client a JavaScript library that provides a WebSocket-like object in the browser. SockJS gives you a coherent, cross-browser, Javascript API which creates a low latency, full duplex, cross-domain communication
Small library that provides functions to tokenize a string into an array of words with or without punctuation
Small, fast, event-driven, fault-tolerant html tokenizer. Works in node or browsers.
Font-Awesome Sass
Lexer / tokenizer
A wide purpose tokenizer for node.js which looks like a stream
WebAssembly port of CRuby with WASI
Use webpack to manage app-like JavaScript modules in Rails
Wix Restaurants credit-cards tokenizer
Simple git hooks manager
A configurable parser to parse whatever you want
llama2 tokenizer for NodeJS/Browser
TS tokenizer for Mistral-based LLMs
gpt4o tokenizer for NodeJS/Browser
Tests for CSS Tokenizers