Source code tokenizer powered by vscode
Tokenize CSS
Tokenized zip support
A promise based streaming tokenizer
Algorithms to help you parse CSS from an array of tokens.
Generic source code tokenizer
A tokenzier for Sass' SCSS syntax
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
TypeScript definition for strtok3 token
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
tokenizer of source code for jscpd
Solve CSS math expressions
Common token types for decoding and encoding numeric and string values
Parses and stringifies CSS selectors
ProseMirror Markdown integration
Tokenizes a string that represents a regular expression.
detector of copy/paste in files
Parse CSS media query lists.
Tokenize a shell string into argv array
r/w stream of glsl tokens
Claude tokenizer
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
Parse CSS Cascade Layer names.
core functionality of copy/paste detector for jscpd
Ember-auth token strategy adapter source code wrapper for ruby libs.
Tokenizes and parses CLDR plural rules and provides a mechanism for emitting them as source code
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Source code lexer configurable for any programming language that allows to tokenize and abstract a given source file
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Saikuro is a Ruby cyclomatic complexity analyzer. When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
When given Ruby source code Saikuro will generate a report listing the cyclomatic complexity of each method found. In addition, Saikuro counts the number of lines per method and can generate a listing of the number of tokens on each line of code.
Credentials like passwords, access tokens and other secrets are often passed to sites each by it's own ENV variable. This is both uncool, non-atomic and therefore unreliable. Use this plugin to store your credentials in encrypted YAML files which you can safely commit to your source code repository. In order to use all of them in Bridgetown, you have to set or pass exactly one ENV variable holding the key to decrypt.
A Ruby FFI binding to the `sas-lexer` Rust crate by Misha Perlov (https://github.com/mishamsk/sas-lexer). Tokenizes SAS source code into a stream of typed tokens with full position metadata. Ships prebuilt native libraries for supported platforms; a runtime FFI loader picks the matching one for the host.
The **dropbox-api** is a Ruby gem for managing DropBox uploading, downloading and sharing DropBox files and folders. The main goals of building this gem are: 1. Simulate a **permanent access token**, since [Dropbox is moving to "short-term live access codes"](https://www.dropboxforum.com/t5/Discuss-Dropbox-Developer-API/Permanent-access-token/td-p/592644); 2. Manage DropBox as an elastic-storage providers for [our SaaS projects](https://github.com/leandrosardi/mysaas), allowing us to upload, download and share download links to files; 3. Backup and restore secret files of our projects that cannot be commited into the source code repository (E.g.: database passwords, SSL certificates, private keys).