Results for gpt_tokenizer

Rust BPE Encoder Decoder (Tokenizer) for GPT-2 / GPT-3

Count tokens for LLM models using exact tokenization

The most complete tokenizer library in Rust - BPE, WordPiece, Unigram, with native support for GPT, BERT, Llama, Claude

Fast token count estimation for LLMs at 96% accuracy without a full tokenizer

Flatten codebases into AI-friendly format

Parallel CLI that counts GPT-style tokens across your codebase

HPC Rust LLM Tokenizer Library

A CLI tool to create commit messages with OpenAI GPT-5.4 for Git and Jujutsu (JJ) repositories

Reusable AI assistant library for local LLM integration (Ollama, LM Studio, etc.)

elizaOS OpenAI Plugin - OpenAI API integration for text, image, audio, and embeddings

LLM tokenizer library with caching and chat template support

Library for encoding and decoding with the tiktoken library in Rust