Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Short summary

Gigatoken is an MIT-licensed Rust BPE tokenizer achieving 24.53 GB/s encoding on a 144-core AMD EPYC 9565, making it 989x faster than HuggingFace tokenizers and 681x faster than tiktoken. The speed gains come from a hand-written SWAR pretokenizer and pretoken caching rather than optimizing the BPE merge loop itself. It targets GPT-2 tokenization but the approach could extend to other BPE vocabularies.

• •Gigatoken hits 24.53 GB/s tokenization on 144-core AMD EPYC 9565
• •989x faster than HuggingFace tokenizers, 681x faster than tiktoken
• •Key innovation is SWAR pretokenizer and pretoken caching, not BPE merge optimization

Generated with AI, which can make mistakes.