Understanding Kafka as a distributed log: implications for consumers, replay, and scaling

Short summary

The article reframes Kafka as an append-only log rather than a queue, explaining how this fundamental difference unlocks independent consumer reads, offset-based replay, and horizontal scaling via partitions. Traditional queues delete messages on consumption; Kafka retains them, enabling multiple services to read the same stream independently and replay history on demand.

• •Kafka is a durable log, not a temporary queue — messages persist independently of consumption
• •Multiple consumer groups read the same stream at their own offsets, enabling replay and parallel processing
• •Partitions scale throughput horizontally; exactly-once delivery is about offset commit semantics, not magic

Generated with AI, which can make mistakes.