Results for tokn

Working on some new ideas for LLM tokenization. WIP, coming soon...

Unified wrapper crate that re-exports the tokn Rust workspace crates

CLI and local gateway runtime for serving OpenAI-compatible APIs via tokn

Account pooling, affinity, and model route resolution across tokn providers

Authentication provider interfaces and credential storage for tokn providers

Model and provider catalogue loading and mapping utilities for tokn

Configuration loading and path resolution for tokn

Request and response conversion pipeline across tokn endpoint formats

Core types, traits, provider contracts, and shared utilities for tokn

Shared endpoint primitives and traits used by tokn endpoint crates

Procedural macros used by tokn endpoint schema crates

Header schema, parsing, and persona utilities shared across tokn crates

Shared test fixtures for tokn-router workspace

Given a script containing token descriptions (each a regular expression), tokn compiles an automaton which it can then use to efficiently convert a text file to a sequence of those tokens.