Universal pre-token language adaptation layer for text-based LLMs
A string pre-tokenizer for C-like syntaxes.
Fast, zero-allocation pretokenizers for BPE tokenizers
Sparse linear algebra solvers
Tokenizers for TrustformeRS