Multilingual tokenizer that automatically tags each token with its type
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
Tokenizer for OpenAI large language models.
gemma3 tokenizer for NodeJS/Browser
Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle
Tiny JavaScript tokenizer.
Shows code error fragment of input file
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
safely inject a block of tokens into a shader
Parse CSS color values
<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
A primitive to tokenize your solid-components to enable custom parsing.
Determine the scope depth of an array of GLSL tokens
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
Tokenize Excel formulas
Trim the whitespace within an array of GLSL tokens
This is an internal library. If you just want to lint your pug templates, use [eslint-plugin-vue-pug](https://github.com/rashfael/eslint-plugin-vue-pug).
Infer the scope of each token in an array of GLSL tokens
Retrieve the values defined with preprocessor statements in a selection of GLSL tokens
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
JS tokenizer for LLaMA-based LLMs