Tokenizes a text from left to right and returns the list of tokens with their types.
JS/WASM bindings for tiktoken
Fast CBOR with a focus on strictness
JS/WASM bindings for tiktoken
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
Parse simple expressions, in a language of your own description
JavaScript port of tiktoken
Hunspell compatible spell checker
A simple string tokeniser
JavaScript port of tiktoken
search and replace tokens from a source file, writing to a new file
Taiwanese Hokkien Transliterator and Tokeniser
JavaScript port of tiktoken
ECMAScript AST recursive visitor
Array#isArray for older browsers
deterministic `JSON.stringify()` - a faster version of substack's json-stable-strigify without jsonify
Light ECMAScript (JavaScript) Value Notation - human written, concise, typed, flexible
Buffers events from a stream until you are ready to handle them.
Get the command from a shebang
CLI arguments parser. Native port of python's argparse.
Resolve the path of a module like `require.resolve()` but from a given path
JSON parse & stringify that supports binary via bops & base64
Run a function exactly one time
Callback wrapping utility