When you send a message to an AI, the model does two distinct things in sequence.

The first is prefill: the model reads your entire prompt at once and builds a set of internal representations called the KV cache. This is computationally intense but highly parallelizable — it happens fast, in a single pass.

The second is decode: the model generates your response one token at a time, each word depending on everything before it. This phase is inherently sequential and must constantly read and update the KV cache as the response grows. That makes it memory-bandwidth-bound — the bottleneck is how fast the chip can move data, not how fast it can compute.

This split drives real infrastructure decisions. Chips designed for inference obsess over memory bandwidth because decode is where most serving time is spent. It also explains why longer responses feel slower — every additional token is another trip through the decode loop.