Generating text one token at a time: keep every earlier key and value instead of computing them again. Prefill against decode, the work and the memory, and a sliding window that evicts the oldest.
This interactive visualization needs JavaScript.