Kog hits 3K t/s on MI300X, no kernel switches — test it now

Short summary

Kog AI's monokernel approach eliminates per-token GPU kernel launches on AMD MI300X, achieving 3,000+ tokens/s for 2B models at batch size 1 by collapsing the entire decode loop into a single persistent kernel. The optimization saves ~4.5 μs per kernel launch and ~7 μs in synchronization overhead by using sentinel-value polling instead of atomics. Test it free via playground.kog.ai or replicate using hand-written HIP/assembly.

• •Monokernel collapses entire LLM decode (prefill, decode, sampling, EOS check) into one GPU kernel, eliminating per-token kernel launch overhead
• •Achieves 3,000+ tokens/s on 8× MI300X node for 2B FP16 models at batch 1; synchronization optimization cuts polling latency from 7.8 μs to 0.9 μs
• •Free playground at playground.kog.ai lets you test immediately; full HIP replication requires assembly-level GPU expertise

Generated with AI, which can make mistakes.