Results for simple_token

This's a run-time loadable extension of SQLite fts5, supports Chinese and pinyin word segmentation and search.

A simple LSP that tokenizes all open files and provides all words as completions. Useful for prose, of languages that use strings like enums (eg: typescript, javascript)

A tiny no_std tokenizer with line & column tracking.

Simple cancelation token

Simplified Version of the SPL Token.

A simple token list representation

Convert code repositories into LLM-friendly prompts with smart chunking and filtering

simple JWT encoder/decoder

A simple cancellation token implementation for Smol, copied from tokio-util's cancellation token, and replace the `waker` with `EventListener` to make it cross async runtime compatible.

A simple tokenizer

This crate provides a way to simply set up a tokenizer and use it. Not recommended for simple tokenizers as this crate adds a bunch of stuff to support many if not all kinds of tokenizers.

Core BitNet implementation with fundamental data structures and algorithms