Read a file, tokenize it, and spit out a handy JSON.
Efficiently modify strings containing ANSI escape codes
Transform stream that tokenizes CSS
Tokenize a string into an array of string parts and format identifier objects.
A tokenzier for Sass' SCSS syntax
transform stream to tokenize html
`@kt3k/tku` is a CLI tool that counts the total number of tokens in a git repository. It uses [tiktoken](https://www.npmjs.com/package/tiktoken) to tokenize file contents and reports the token count per file and in total.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parse CSS color values
Tokenize a string.
Tokenize CSS
Provide a high level wrapper for kuromoji.js
Tokenize Excel formulas
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
A drop-in replacement for react-markdown, designed for AI-powered streaming.
The lexer for Materialize's SQL dialect, with wasm build targets.
Skyflow SDK for Node.js
Small library that provides functions to tokenize a string into an array of words with or without punctuation
micromark utility to tokenize subtokens
Multilingual tokenizer that automatically tags each token with its type
Estimate the number of tokens for Gemini models
ProseMirror Markdown integration
match a tokenized html stream with css selectors
tokenize a string that includes ansi code
Given a script containing token descriptions (each a regular expression), tokn compiles an automaton which it can then use to efficiently convert a text file to a sequence of those tokens.
Given a config file and a github token, checks for open PRs
TokenEstimator is a Rails gem that allows you to count tokens in Excel, CSV, PDF, TXT, Markdown, and input text files using different tokenizers.
Asciidoctor extension for including files from private GitHub repos
Given a master vault token, issue short-lived, per-application tokens to each app in a docker-compose.yml file, restricting each app the to corresponding security policy.
A Ruby gem for code signing operations using hardware tokens (HSM/smart cards) via PKCS#11. Supports SafeNet eToken, RFC 3161 timestamping, PDF visible signatures, and signature verification.
Logentries output plugin for Fluent event without Logentries config file, just with simple token
Given a gitlab personal token (https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) and a gitlab project, produce a DOAP (https://github.com/ewilderj/doap/wiki) XML file.
Source code lexer configurable for any programming language that allows to tokenize and abstract a given source file
Takes tailed files in a single tail call and sends them through to Heroku Logplex with custom logplex tokens.
🪙 Token::Resolver provides configurable PEG-based (parslet) parsing and resolution of structured tokens (e.g., {KJ|GEM_NAME}) in arbitrary text. Useful for template ETL pipelines where tokens in template files must be resolved before format-specific merging.
Stockade is a lexer that reads unstructured text information (from files, logs, databases etc.) and tokenizes pieces that look like personally identifiable information (PII).