Strav view engine — .strav template language. Tokenizer + compiler + ViewEngine, Vue 3 hydration islands + buildIslands, pages auto-router, console commands, disk cache, asset versioning.
JSON Transform language tokenizer (and syntax highlight), hover provider and more
tokenizer of source code for jscpd
JSON Transform language tokenizer (and syntax highlight), hover provider and more
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Algorithms to help you parse CSS from an array of tokens.
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
ProseMirror Markdown integration
Multilingual tokenizer that automatically tags each token with its type
Common token types for decoding and encoding numeric and string values
Claude tokenizer
r/w stream of glsl tokens
Tokenizes a string that represents a regular expression.
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Tokenize a shell string into argv array
Parse CSS media query lists.
Tools for processing polish language. Tokenization, scanning, categorization...
A Pratt parser. Create token objects to define your language. Create a lexer to return tokens. Call the parser to grok the language.
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
High-level Ruby bindings to the Stanford CoreNLP package, a set natural language processing tools that provides tokenization, part-of-speech tagging and parsing for several languages, as well as named entity recognition and coreference resolution for English, German, French and other languages.
Thai language tools for Ruby, i.e. a word tokenizer, a character level indentifier, and a romanization tool
TOON is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage.
Source code lexer configurable for any programming language that allows to tokenize and abstract a given source file
Textoken is a Ruby library for text tokenization. This gem extracts words from text with many customizations. It can be used in many fields like Web Crawling and Natural Language Processing.
Auth0 (https://auth0.com) is web service handling users identities which can be easily plugged into your application. It provides SDKs for many languages which enable you to sign up/in users and returns access token (JWT) in exchange. Access token can be used then to access your's Web Service. This gem helps you to verify (https://auth0.com/docs/api-auth/tutorials/verify-access-token#verify-the-signature) such access token which has been signed using the RS256 algorithm.
A Ruby gem that converts T::Struct and T::Enum to BAML (Boundary AI Markup Language) type definitions. BAML uses 60% fewer tokens than JSON Schema while maintaining type safety.
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.