a lightweight no-dependency fork of transformers.js (only tokenizers)
TypeScript definitions for css-selector-tokenizer
<!-- :begin use tokenizer/banner -->
A function to parse floating point hexadecimal strings as defined by the WebAssembly specification
<!-- :begin use tokenizer/banner -->
Create and animate hand-drawn annotations on a web page
Precompile Handlebars templates to JST file.
<!-- :begin use tokenizer/banner -->
JavaScript implementation of Japanese morphological analyzer
Curried property accessor function that resolves deeply-nested object properties via dot/bracket-notation string path while mitigating TypeErrors via friendly and composable API.
Trim the whitespace within an array of GLSL tokens
asn1js is a pure JavaScript library implementing this standard. ASN.1 is the basis of all X.509 related data structures and numerous other protocols used on the web
Retrieve the values defined with preprocessor statements in a selection of GLSL tokens
🐊Putout plugin adds ability to remove useless array inside property accessors
Object readings and transformations using dot notation syntax
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
Tokenizer utilities for Tokenlens.
<!-- :begin use tokenizer/banner -->
cjson - Commented JavaScript Object Notation. It is a json loader, which parses only valid json files, but with comments enabled. Useful for loading configs.
JS tokenizer for LLaMA 3
This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or
gemini tokenizer for NodeJS/Browser
"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.
Infer the scope of each token in an array of GLSL tokens