Converts text into tokens, as defined by provided syntax; Supports recursive sub-syntaxes, regular expressions, and more..
Build &str tokenizers using regex
Fast byte-pair encoding implementation.
Rust SDK for ChmlFrp