What are KV caches really?
KV caching lies sufficiently deep within the guts of how transformers work that really understanding how it can be used goes beyond any simple analogy. This is why the Internet is full of explanations of KV caching that are either grossly oversimplified (and therefore imply that KV caching is more useful than it really is) or dive straight into mathematical formalism (and therefore are incomprehensible to dumb-dumbs like me). So I will try to thread that needle and explain how KV caching works in terms of how dumb-dumbs like me actually experience them. What are keys, values, and KV caches? Keys, values, and KV caches only make sense if you first understand what's happening when you ask a chatbot a question. There are fundamentally two steps to inferencing: prefill and decode. And anyone who's used ChatGPT or Claude has seen them firsthand. What is prefill? This is prefill : When you ask a chatbot a question, there will be a lag before output starts being spit out. ...
