r/w stream of glsl tokens
Claude tokenizer
Tokenizes a string that represents a regular expression.
Parse CSS media query lists.
detector of copy/paste in files
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Multilingual tokenizer that automatically tags each token with its type
Retrieve the values defined with preprocessor statements in a selection of GLSL tokens
gemma3 tokenizer for NodeJS/Browser
Parse CSS Cascade Layer names.
core functionality of copy/paste detector for jscpd
Extract function definitions from an array of GLSL tokens.
This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
Shows code error fragment of input file
Lexer / tokenizer
Parse CSS color values
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Take an array of GLSL tokens and determine which tokens are either assignments or variable declarations.
<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
Provide a high level wrapper for kuromoji.js
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
JS tokenizer for LLaMA-based LLMs
A primitive to tokenize your solid-components to enable custom parsing.