Turns a string into tokens.
Efficiently modify strings containing ANSI escape codes
Tokenize a string into an array of string parts and format identifier objects.
Transform stream that tokenizes CSS
A tokenzier for Sass' SCSS syntax
transform stream to tokenize html
Tokenize a string.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parse CSS color values
Tokenize CSS
micromark utility to tokenize subtokens
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
Tokenize Excel formulas
Provide a high level wrapper for kuromoji.js
Small library that provides functions to tokenize a string into an array of words with or without punctuation
match a tokenized html stream with css selectors
tokenize a string that includes ansi code
The lexer for Materialize's SQL dialect, with wasm build targets.
ProseMirror Markdown integration
Multilingual tokenizer that automatically tags each token with its type
Parser for edge template engine
Estimate the number of tokens for Gemini models
Tokenize a text into sentences
Skyflow SDK for Node.js
HTML Tokenizer
A pure ruby implementation of the RFC 7519 OAuth JSON Web Token (JWT) standard.
Google ID Token utilities; currently just a parser/checker
JWT token generator extension for Doorkeeper
A multilingual tokenizer to split a string into tokens.
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
A library for generating signed authentication tokens for use with Firebase. Uses your app secret.
Token authentication for Devise which supports multiple tokens per model
High speed text tokenization for Ruby
Simple gem for decoding JWT token
Design Tokens for the Polaris Design System
A ruby implementation of the parsing algorithm in HTML5. == FEATURES/PROBLEMS: == SYNOPSIS: TODO