Additional tokenizers for Orama
Multi-arch builds of HuggingFace tokenizers
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Multi-arch builds of HuggingFace tokenizers
a lightweight no-dependency fork of transformers.js (only tokenizers)
Multi-arch builds of HuggingFace tokenizers
Multi-arch builds of HuggingFace tokenizers
| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |
Fork of `tokenizers`
Multi-arch builds of HuggingFace tokenizers
Multi-arch builds of HuggingFace tokenizers
Tests for CSS Tokenizers
Multi-arch builds of HuggingFace tokenizers
Multi-arch builds of HuggingFace tokenizers
A markdown parser with rich built-in tokenizers
| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |
Multi-arch builds of HuggingFace tokenizers
The API for building streaming tokenizers and lexers.
Tokenizers platform specific bindings
基于 [HuggingFace Tokenizers](https://github.com/huggingface/tokenizers) 的JavaScript封装,使用WebAssembly实现高性能的文本标记化功能。
Tokenizers platform specific bindings
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Tokenizers, parsers and expression calculators for Pip.Services in Node.js / ES2017
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Hermit incentive contract
Full-featured JWT authentication middleware for actix-web: login, logout, refresh, token rotation, cookie management, RSA/HMAC, RBAC authorizer, pluggable token store
A parser tool to generate recursive descent top down parser.
A powerful Rust authentication and authorization framework
Auto-generated client library for the Edgegap API, used by the arbctl tool
This is the SDK for Rust. Like all BlockChyp SDKs, it provides a full client for the BlockChyp gateway and BlockChyp payment terminals.
Hessra biscuit token SDK for Rust
Core library for sa-token-rust, a powerful authentication and authorization framework
Rust SDK for Nad.fun
Secure, typed, async Rust SDK for OpenBao
Implementation of the Common Access Token (CAT) specification
Fast state-of-the-art tokenizers for Ruby
A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
HTML Tokenizer
A pure ruby implementation of the RFC 7519 OAuth JSON Web Token (JWT) standard.
Google ID Token utilities; currently just a parser/checker
JWT token generator extension for Doorkeeper
A multilingual tokenizer to split a string into tokens.
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
A library for generating signed authentication tokens for use with Firebase. Uses your app secret.
Token authentication for Devise which supports multiple tokens per model
High speed text tokenization for Ruby
Design Tokens for the Polaris Design System
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.