gemma3 tokenizer for NodeJS/Browser
Multilingual tokenizer that automatically tags each token with its type
Tokenizer for OpenAI large language models.
Parse CSS color values
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
core functionality of copy/paste detector for jscpd
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Shows code error fragment of input file
safely inject a block of tokens into a shader
<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
A primitive to tokenize your solid-components to enable custom parsing.
My JavaScript parser
Range-request tokenizer adapter
This is an internal library. If you just want to lint your pug templates, use [eslint-plugin-vue-pug](https://github.com/rashfael/eslint-plugin-vue-pug).
Tokenize Excel formulas
Retrieve the values defined with preprocessor statements in a selection of GLSL tokens
Trim the whitespace within an array of GLSL tokens
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->