Next-Token Prediction: How LLM Text Generation Actually Works

Short summary

LLMs generate text by repeatedly predicting a probability distribution over the entire vocabulary for the next token, then sampling from it — not deterministically picking the top result. This sampling mechanism explains why identical prompts yield different outputs, how context reshapes predictions, and why compute cost scales per token rather than per reply. Greedy decoding (temperature 0) trades variety for reproducibility.

• •LLMs compute a fresh probability distribution over all possible next tokens and sample from it, not always picking the top score
• •Sampling explains non-deterministic outputs and how context changes predictions without changing the model
• •Compute cost is per token — a 500-token response costs ~500x more than a 1-token response

Generated with AI, which can make mistakes.