Training vs Inference: Why Building Costs Millions and Asking Costs Cents

In the last episode, we saw how embeddings turn words into numbers that capture meaning. That is the model’s internal language. But the model itself had to learn those embeddings, along with everything else, during a phase called training. The moment you ask it for a recipe, it’s using a completely different phase: inference. These two phases are often confused, but they are as different as building a car and driving it.

Here is the twist: for a popular chatbot, the total cost of inference over its lifetime often dwarfs the training cost. That changes how you think about what is “expensive.” The million-dollar training run gets the headlines. The pennies per query add up to millions every year, silently.

What actually happens when you train a large language model?

Training is a massive optimization problem. The model sees billions of example sentences, tries to predict the next word, measures how wrong it is, and adjusts its billions of parameters to get better. Think of it like a student reading the entire internet. After every sentence, the student takes a pop quiz: “What word comes next?” They check the answer, note every mistake, and update their mental rules. That loop runs for weeks on thousands of GPUs.

During training, the model processes a batch of token sequences. It runs a forward pass through all its layers to produce a prediction for each position. Then it computes a loss, usually cross-entropy, between its predictions and the real next tokens. The backward pass calculates how every parameter contributed to the error. Finally, an optimizer like Adam updates each parameter to reduce the loss next time. This forward-backward-optimizer cycle is one training step.

The math behind the cost is straightforward. For a transformer with P parameters, one forward pass costs about 2P floating-point operations (FLOPs) per token. Adding the backward pass and optimizer update brings the total to roughly 6P FLOPs per token. GPT-3 has 175 billion parameters and was trained on about 300 billion tokens. That works out to around 3.15×10²³ FLOPs, or 3,640 petaflop-days. No single GPU (graphics processing unit) can run that in a human lifetime. Training must be distributed across clusters of hundreds or thousands of accelerators.

Distributed training uses several tricks. Data parallelism gives each GPU a full copy of the model and a different slice of data. After each step, GPUs average their gradients. Tensor parallelism splits individual weight matrices across GPUs, so each device only stores a shard. Pipeline parallelism divides layers into stages, passing micro-batches through like an assembly line. Systems like Megatron-LM and DeepSpeed ZeRO combine all three to fit enormous models into limited memory and keep GPUs busy. The cluster behaves like one giant computer, bound by network bandwidth as much as by raw FLOPs.

So the model you chat with was not born knowing about chocolate cake. It read millions of recipes, forum posts, and cookbooks during training. It learned patterns like “butter, sugar, flour” appearing together. That education cost millions of dollars and took months.

What happens when you ask a model a question?

Inference is the model applying what it learned to a new input. It processes your prompt in a single forward pass, then generates one token at a time. The key trick that makes this fast is a KV (key-value) cache. It avoids recomputing attention for tokens it has already seen.

You open a chatbot and type, “Give me a recipe for chocolate cake.” The system tokenizes your text into a sequence of token IDs. Then it enters the prefill phase. All input tokens go through the model at once. For each token at each layer, the model computes key and value vectors and stashes them in the KV cache. At the end, it produces a logit for the first new token. This prefill step is compute-heavy but happens only once per prompt.

Now the decode phase begins. The model takes that first generated token, embeds it, and runs a forward pass that reads the stored keys and values from the cache. It only computes new key, value, and query vectors for the current token. It attends over the entire prefix without recomputing anything. The output is a logit for the next token. The process repeats, “Sure!”, “Here”, “is”, “a”, “simple”, “recipe”, until a stop token appears.

Each decode step still reads the entire model’s weights from GPU memory. But the math per byte of weight data is tiny. This makes the decode phase memory-bandwidth-bound. The GPU’s compute units spend most of their time waiting for data from HBM. That is why inference hardware prizes memory bandwidth over peak FLOPs.

The KV cache grows with sequence length. For a model with L layers, H attention heads, and head dimension d, each token adds 2 × L × H × d numbers to memory. For long conversations, the cache can rival the model weights in size. Managing that memory is a central challenge of serving.

Why does training cost millions while inference costs cents?

Training requires thousands of GPUs running for weeks. Each token processed triggers a full forward and backward pass plus optimizer state updates. The total FLOPs are enormous. Electricity, hardware depreciation, and engineering time add up. Estimates for GPT-3’s training run range from $4 million to $12 million. Larger models like GPT-4 likely cost over $100 million.

Inference, on the other hand, runs a forward-only pass. The cost per token is roughly one-third of a training token’s FLOPs. A single query might involve a few hundred input tokens and a few hundred output tokens. At a typical price of $0.001 per 1,000 tokens, that query costs a fraction of a cent. A single GPU can serve dozens of users concurrently.

But the cents add up. A service handling 100 million queries per day at $0.002 per request spends about $73 million per year on inference. Over a model’s lifetime, inference often dominates total cost. The training bill is a one-time capital expense. Inference is an ongoing operational expense that scales with every new user.

This explains why so much engineering effort in 2024-2026 has focused on inference optimization. Techniques like quantization, speculative decoding, and PagedAttention squeeze more tokens per second out of the same hardware. They attack the memory bandwidth and KV cache bottlenecks directly.

Why is inference more about memory bandwidth than compute?

During decode, the model reads every one of its billions of weights from GPU memory for each token it generates. The arithmetic intensity, FLOPs per byte of data moved, is low. A modern GPU like an H100 can perform over a thousand trillion FLOPs per second, but its memory bandwidth is “only” 3.35 terabytes per second. The math finishes quickly, and the GPU stalls waiting for the next chunk of weights.

Think of a librarian fetching books from miles of shelves. The bottleneck is how fast they can walk, not how fast they can read the title. For inference, faster memory (higher bandwidth HBM) directly translates to more tokens per second. That is why the H200, with its larger and faster memory, outperforms the H100 on decode-heavy workloads even though peak FLOPs are similar.

Prefill is different. It processes many tokens in parallel, doing large matrix multiplications. There, compute throughput matters. But for chat, where output tokens outnumber input tokens, decode dominates. Memory bandwidth is king.

How do serving systems handle multiple users at once?

A naive approach would process one request at a time. The GPU would sit idle during the memory-bound decode steps. Modern serving engines use continuous batching. They group requests that arrive at different times into a single batch. When one request finishes, a new one can join immediately. This keeps the GPU fed with enough work to hide memory latency.

They also manage KV caches intelligently. PagedAttention, introduced by the vLLM project, treats the KV cache like virtual memory. It allocates cache in blocks and maps them to sequences, avoiding fragmentation and allowing memory sharing when prompts share a prefix. This increases the number of concurrent users a single GPU can serve.

These systems turn a handful of GPUs into a service that handles thousands of simultaneous conversations. The same model that required a supercomputer to train now runs on a commodity server, answering recipe requests for pennies.

Quick Reference

Frequently Asked Questions

Q: Does fine-tuning count as training or inference? Fine-tuning is training, just on a smaller scale. It runs forward and backward passes to update the model’s weights, using a fraction of the original compute. The cost is far lower than pretraining but still uses training infrastructure.

Q: Why can’t we use the same hardware for both training and inference? You can, but it is usually wasteful. Training needs high-bandwidth interconnects between thousands of GPUs and favors peak FLOPs. Inference benefits from high memory bandwidth and smaller, cheaper clusters. Dedicated inference hardware like the L40S or H200 is optimized differently.

Q: How does the KV cache affect latency? The KV cache grows linearly with context length. For very long conversations, reading and updating the cache can become a bottleneck. If the cache exceeds GPU memory, the system must spill to CPU (central processing unit) or disk, causing latency spikes. Efficient cache management is critical for long-context serving.

Q: What is the biggest cost driver in a production LLM service? For most chat applications, the decode phase dominates. Output tokens are generated one by one, each requiring a full model weight read. Memory bandwidth and the number of concurrent users determine how many GPUs you need. Reducing per-token cost through quantization and batching has the largest impact on total spend.

Q: Can inference be done on CPUs? Yes, but it is slow for large models. CPUs have much lower memory bandwidth than GPUs. A model that generates 20 tokens per second on a GPU might manage 1-2 tokens per second on a high-end CPU. For latency-sensitive chat, GPUs are necessary. For batch processing where latency is less critical, CPUs can be cost-effective.

Test yourself

You are building a customer support chatbot that handles 10,000 conversations per day. Each conversation averages 500 input tokens and 200 output tokens. Your model provider charges $0.001 per 1,000 tokens for inference. What is the daily compute cost? How would you reduce it if the service grew 100x?

Answer: Total tokens per conversation: 500 + 200 = 700 tokens. Daily tokens: 10,000 × 700 = 7,000,000 tokens. Cost: 7,000 × $0.001 = $7.00 per day. At 100x growth, that becomes $700 per day, or about $255,000 per year. To reduce costs, you could switch to a quantized model (often half the cost with minimal quality loss), implement semantic caching so identical or similar questions reuse previous answers, or batch requests during off-peak hours to get volume discounts. You might also fine-tune a smaller model on your support data so it can run on cheaper hardware, or use speculative decoding to generate multiple tokens per step, cutting the number of forward passes.

If you want this kind of breakdown every week, how real AI systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Next time, we will look at how models are fine-tuned to follow instructions. That is the step that turns a raw text predictor into a helpful assistant.

Sources

• GPT-3 paper (training compute, architecture)
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
• Lambda Labs GPU benchmarks and cost estimates
• LLM inference performance analysis (memory bandwidth bound)