Results for polyglot_tokenizer

Tokenizer for world's most spoken languages and social media texts like Facebook, Twitter etc.

A generic programming language tokenizer

A generic programming language tokenizer

Lexical analysis and polyglot tokenization for Ling

Context-native workspace manager for AI coding agents. Cuts token consumption on polyglot monorepos.

SQL parsing, validating, formatting, and dialect translation library

Internal crate for kdo — tree-sitter signature extraction and token-budgeted context generation. Not intended for direct use; API may change without notice.

duckman - a DuckDB version manager and toolchain CLI

Internal crate for kdo — core types and errors. Not intended for direct use; API may change without notice.

Internal crate for kdo — workspace dependency graph, discovery, traversal, content hashing. Not intended for direct use; API may change without notice.

Internal crate for kdo — manifest parsers (Cargo, Node, Python, Anchor, Go). Not intended for direct use; API may change without notice.

Internal crate for kdo — MCP server exposing workspace intelligence over JSON-RPC/stdio. Not intended for direct use; API may change without notice.

ASIMOV Software Development Kit (SDK) for Rust