A wide purpose tokenizer for node.js which looks like a stream
Tiny JavaScript tokenizer.
A promise based streaming tokenizer
Lexer / tokenizer
tokenizer of source code for jscpd
Claude tokenizer
gemma3 tokenizer for NodeJS/Browser
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Range-request tokenizer adapter
Amazon S3 tokenizer
Multilingual tokenizer that automatically tags each token with its type
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
Tokenizer for processing admonitions
JSON AST parser, tokenizer, printer, traverser.
Tokenizer primitives of Shiki
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
CSS / Extended CSS tokenizer
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Thai text tokenizer
Hermit incentive contract
Full-featured JWT authentication middleware for actix-web: login, logout, refresh, token rotation, cookie management, RSA/HMAC, RBAC authorizer, pluggable token store
A parser tool to generate recursive descent top down parser.
A powerful Rust authentication and authorization framework
Auto-generated client library for the Edgegap API, used by the arbctl tool
This is the SDK for Rust. Like all BlockChyp SDKs, it provides a full client for the BlockChyp gateway and BlockChyp payment terminals.
Hessra biscuit token SDK for Rust
Core library for sa-token-rust, a powerful authentication and authorization framework
Rust SDK for Nad.fun
Secure, typed, async Rust SDK for OpenBao
Implementation of the Common Access Token (CAT) specification
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
HTML Tokenizer
A multilingual tokenizer to split a string into tokens.
Gem that wraps up the the tokenizer cores
Tokenize url by variety of providers.
TactfulTokenizer uses a naive bayesian model train on the Brown and WSJ corpuses to provide high quality sentence tokenization.
LLT's Tokenizer
Tokenizes strings for use in social applications.
RubyTokenizer is a simple language processing command-line tool. It performs low-level tokenization and returns the top 10 most frequent words in a body of text. At the moment it's only available for English texts and it segments words by filtering whitespaces, punctuation marks, parantheses and other special characters.
ModelTokenizer creates random tokens to be used as primary keys for ActiveRecord objects
Tokenize English, Dutch, German, Italian and Spanish to KAF
tokenizer
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.