<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
<!-- :begin use tokenizer/banner -->
JS tokenizer for Mistral-based LLMs
Tokenize Excel formulas
Tokenizer for processing inline math (formulas)
<!-- :begin use tokenizer/banner -->
This is an internal library. If you just want to lint your pug templates, use [eslint-plugin-vue-pug](https://github.com/rashfael/eslint-plugin-vue-pug).
<!-- :begin use tokenizer/banner -->
Tokenizer for processing fenced math block (formulas)
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Error-tolerant ECMAScript parser
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Tokenizer for processing admonitions
<!-- :begin use tokenizer/banner -->
Range-request tokenizer adapter
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
<!-- :begin use tokenizer/banner -->
Easy-to-use plugin system for creating powerful, fast and versatile parsers and compilers, with built-in source-map support.