Series · 7 of 7 parts · Advanced
Inference Internals
What actually happens between hitting enter and seeing tokens: the caches, the batching, and the economics under every AI product.
What you’ll understand
By the end you understand serving well enough to cut costs, explain latency, and read model pricing like an engineer.
Episodes
01The Life of a Request: What Happens When You Hit EnterTokenization, prefill, decode, stream: the full path from your prompt to the first word back.9 min→02The KV Cache: Why Long Chats Get Slow and ExpensiveThe memory trick that makes generation fast, and the bill it quietly runs up.13 min→03Batching and PagedAttention: How vLLM Serves Thousands at OnceContinuous batching and paged memory: the two ideas behind modern serving throughput.11 min→04Quantization: Smaller Weights, Same Answers (Mostly)INT8, INT4, and what you actually lose when you shrink a model.10 min→05Speculative Decoding: A Small Model Drafts, a Big Model ApprovesThe clever trick making frontier models feel twice as fast.12 min→06The Economics of Inference: Why API Prices Are What They AreGPUs, utilization, and margins: the cost structure behind per-token pricing.10 min→07Local Inference: Running Models on Your Own MachineOllama, llama.cpp, and what hardware actually gets you usable speed.11 min→