This is a utility library to tokenize any source code into syntax and manipulate on syntax tree.
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
Algorithms to help you parse CSS from an array of tokens.
TypeScript definition for strtok3 token
Solve CSS math expressions
A tokenzier for Sass' SCSS syntax
Parses and stringifies CSS selectors
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
tokenizer of source code for jscpd
Common token types for decoding and encoding numeric and string values
r/w stream of glsl tokens
ProseMirror Markdown integration
Tokenizes a string that represents a regular expression.
Claude tokenizer
Parse CSS media query lists.
Tiny JavaScript tokenizer.
detector of copy/paste in files
Tokenize a shell string into argv array
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
Parse CSS Cascade Layer names.
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Given a script containing token descriptions (each a regular expression), tokn compiles an automaton which it can then use to efficiently convert a text file to a sequence of those tokens.
This gem is meant for Ruby 1.8.x builds. Ruby 1.9.x comes with a functioning ripper implementation out of the box. This gem is a port of the Ripper 1.9 for Ruby 1.8.x. Ripper was written by Minero Aoki and ported by Loren Segal.
Ember-auth token strategy adapter source code wrapper for ruby libs.
Tokenizes and parses CLDR plural rules and provides a mechanism for emitting them as source code
OpenClacky is a 100% open-source (MIT) AI Agent that matches Claude Code on capability at comparable cost. Built on two years of agentic R&D, it ships with a minimal 16-tool core, near-100% prompt cache hit rate, idle-time auto-compression, and BYOK support for any OpenAI-compatible model. A natural-language Skill system lets you create, evolve, and monetize agent capabilities without writing code, plus first-class integrations with Feishu, WeCom, WeChat, Discord, and Telegram.
Source code lexer configurable for any programming language that allows to tokenize and abstract a given source file
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
No description provided.
No description provided.
No description provided.
No description provided.