Training vs Inference: Why Building Costs Millions and Asking Costs Cents
Two completely different phases of a model's life, constantly confused.
In Embeddings: Meaning as Numbers, you saw how a model turns words into numbers that capture meaning. But the model had to learn those numbers, along with everything else, during a phase called training. The moment you ask it for a recipe, it is in a completely different phase: inference. The two are often confused, but they are as different as building a car and driving it.
For a popular chatbot, the total cost of answering questions over its lifetime can grow larger than the cost of training it. That changes how you think about what is "expensive". The training run gets the headlines. The fractions of a cent per question add up quietly, every day.
What actually happens when you train a large language model?
Training is a huge practice session. The model reads trillions of tokens of text, tries to predict each next token, measures how wrong it was, and adjusts its billions of parameters to do a little better. Think of a student reading a vast library. After every sentence, the student takes a pop quiz: "What word comes next?" They check the answer, notice the mistake, and adjust. For the largest models that loop runs for weeks or months on thousands of GPUs (graphics processing units, the chips that do this work). Meta trained its largest Llama 3 model on up to 16,000 of them (The Llama 3 Herd of Models).
Each training step has four parts, and The Groundwork named all of them. The model runs a forward pass, reading a batch of text through all its layers to make a guess at every position. It measures its loss, how far those guesses were from the real next tokens, the miss you met in Getting It Wrong on Purpose. The backward pass, backpropagation, works out which way every parameter should turn. Then an optimizer, often one called Adam, takes a small step downhill, as in Practice Makes It Smaller.
The cost follows a simple rule. One forward pass takes about 2 floating point operations (FLOPs, single multiplications or additions) for every parameter, for every token. The backward pass takes about twice that again, so training costs about 6 FLOPs per parameter per token (Scaling Laws for Neural Language Models). GPT-3 has 175 billion parameters and was trained on 300 billion tokens. Its paper puts the total at about 3.14 × 10²³ FLOPs, a 3 followed by 23 digits, or 3,640 petaflop/s-days (Language Models are Few-Shot Learners). A single 2020-era V100 GPU would have needed about 355 years for that, by Lambda's estimate.
So training is split across thousands of GPUs. Each one takes a slice of the data or a slice of the model, and they constantly share results, so the whole cluster behaves like one giant computer. Systems such as Megatron-LM (Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism) and ZeRO (ZeRO: Memory Optimizations Toward Training Trillion Parameter Models) were built to organize that split.
So the model you chat with was not born knowing about chocolate cake. It read huge numbers of recipes, forum posts and cookbooks during training, and learned patterns like "butter, sugar, flour" turning up together. That education is where the millions of dollars go.
What happens when you ask a model a question?
Inference is the model using what it learned on something new. It reads your whole prompt in one pass, then writes its reply one token at a time.
One thing never happens during inference: learning. The model's parameters are frozen. Your chat does not teach it anything, and it knows nothing that happened after its training text was collected, its knowledge cutoff, which model makers publish (Anthropic's models overview, for example). It seems to remember earlier messages because the app sends the whole conversation back with every new message (Anthropic's guide to its Messages API). Anything "remembered" across separate chats is the app storing notes and putting them back into the prompt.
You open a chatbot and type, "Give me a recipe for chocolate cake." The app turns your text into tokens. Then comes the first phase, called prefill: all of your tokens go through the model at once. Along the way, the model keeps notes on every token it has processed, called the KV cache, so it never has to redo that work. At the end of prefill, it produces a score for every possible next token, turns the scores into probabilities, and picks the first token of its reply.
Then the second phase, decode, begins. The model adds that first token, runs one more pass using its stored notes, and picks the next token. It repeats, "Sure!", "Here", "is", "a", "simple", "recipe", until it produces a token that means "stop".
Every decode step has to read all of the model's parameters from the GPU's memory, but does very little arithmetic with each one. So generating text is limited mostly by how fast the chip can move data, not by how fast it can calculate. The Life of a Request: What Happens When You Hit Enter walks through both phases in detail, and The KV Cache: Why Long Chats Get Slow and Expensive explains those notes.
Why does training cost millions while inference costs cents?
Training needs thousands of GPUs running for weeks, and every token it reads costs a forward pass, a backward pass and an update. Lambda estimated GPT-3's single training run at about $4.6 million in 2020 cloud prices. The biggest runs have grown far more expensive since: one detailed study estimates that the cost of training the most compute hungry models has grown about 2.4 times a year since 2016, that the chips and the staff for models like GPT-4 and Gemini each cost tens of millions of dollars, and that the largest runs will pass a billion dollars by 2027 if the trend continues (The rising costs of training frontier AI models).
Inference runs the forward pass only, so each token costs about a third of the computing work of a training token (Scaling Laws for Neural Language Models).
A typical question involves a few hundred tokens in and a few hundred out. At $2 per million input tokens and $10 per million output tokens, the list price of a mid range model from OpenAI or Anthropic as of October 2026, 500 tokens each way costs about $0.006, well under a cent. And one server can answer many people at the same time by batching their requests together.
But the cents add up. Picture a service that handles 100 million questions a day at $0.002 each. That is $200,000 a day, or about $73 million a year, every year the service runs. Training is a large bill paid up front, while inference is a running cost that grows with every new user, so over a model's lifetime it can become the bigger of the two.
That is why so much engineering effort in recent years has gone into making inference cheaper. Techniques with names like quantization, speculative decoding and PagedAttention squeeze more tokens per second out of the same hardware, and each has its own piece in the Inference Internals series.
What makes inference fast or slow?
Think of a librarian who must fetch every book in a vast library to answer each question, one word at a time. Reading each title takes no time at all. The walking is what takes time. A GPU generating text is in the same position: the arithmetic for each token finishes quickly, and the chip spends most of its time waiting for the next batch of parameters to arrive from memory. That is why chips for serving models prize fast memory.
Reading your prompt is different. All of its tokens are processed at once in one big parallel pass, so there the raw calculating speed matters. In a typical chat, the reply usually dominates the wait, because output tokens come out one after another, while the whole input went through in a single pass.
Serving companies keep their GPUs busy by batching many users' requests together and by managing the notes each conversation keeps. Batching and PagedAttention: How vLLM Serves Hundreds at Once shows how one popular system does it, and The Economics of Inference: Why API Prices Are What They Are shows how all of this turns into the prices you pay.
Quick Reference
| Property | Training | Inference |
|---|---|---|
| What runs | Forward pass, backward pass, update | Forward pass only |
| Changes the model's parameters? | Yes, that is the point | No, never |
| FLOPs per token | About 6 per parameter | About 2 per parameter |
| Main limit | Raw computing power and memory | Memory speed and the notes each conversation keeps |
| Scale | Thousands of GPUs at once | A few GPUs per copy of the model, many copies |
| Time | Weeks to months per run | Around the clock, for as long as people use it |
| Cost | A large bill up front | A small cost per token that adds up over time |
Frequently Asked Questions
Q: Does fine-tuning count as training or inference? Fine-tuning is training, just on a smaller scale. It runs forward and backward passes to update the model's parameters, using a small fraction of the original computing work. Fine-Tuning, Explained: Teaching an Old Model New Tricks covers it.
Q: Why not use exactly the same setup for training and inference? You can use the same kind of chips, but the setups differ. Training wants a huge number of chips tightly linked so they can share results constantly. Serving wants lots of fast memory and can run each copy of the model on a small group of chips, close to its users.
Q: How do long conversations affect speed? The notes the model keeps, the KV cache, grow with every token in the conversation. Very long conversations take more memory and more reading per new token, and when a server runs short of memory it has to pause some requests and redo their work later, which shows up as delays. The KV Cache: Why Long Chats Get Slow and Expensive explains this.
Q: What is the biggest cost in running a chatbot? Often the replies. Output tokens are produced one at a time, each needing a pass through the whole model, and providers price them several times higher than input tokens: five times higher across OpenAI's and Anthropic's current models, as of October 2026. Long inputs, such as documents or long chat histories, add up too.
Q: Can inference run on an ordinary computer? Yes, for smaller models. Ordinary processors move data far more slowly than GPUs, so large models are slow on them, but small open models run well on laptops and desktops. Local Inference: Running Models on Your Own Machine shows what to expect.
Test yourself
You run a customer support chatbot that handles 10,000 conversations a day. Each averages 500 input tokens and 200 output tokens. Your provider charges $2 per million input tokens and $10 per million output tokens. What does it cost per day, and how would you cut the bill if traffic grew 100 times?
Answer: Input: 10,000 × 500 = 5 million tokens, which costs $10. Output: 10,000 × 200 = 2 million tokens, which costs $20. That is $30 a day. At 100 times the traffic it becomes $3,000 a day, about $1.1 million a year. As a customer paying per token, you control the levers that matter. Route simple questions to a cheaper, smaller model. If your fixed instructions are long enough to be cached, keep them first and identical so prompt caching discounts them. Cap the length of replies, since output costs five times as much here. Use the provider's discounted batch mode, half price at Anthropic, only for work that can wait, such as nightly summaries; live chat cannot. If you run the model yourself, quantization and speculative decoding also help.
Next: Context Windows: The Model's Working Memory. It explains how much a model can take in at once, and why it can miss things long before the limit.
If you want this kind of breakdown every week, how real AI systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- The Llama 3 Herd of Models (Meta 2024)
- Scaling Laws for Neural Language Models (Kaplan et al. 2020), the 2 and 6 FLOPs per parameter rule
- Language Models are Few-Shot Learners (Brown et al. 2020), GPT-3's training compute in Table D.1
- Demystifying GPT-3 (Lambda, June 2020)
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al. 2019)
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al. 2019)
- The rising costs of training frontier AI models (Cottier et al. 2024)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. 2023)
- Models overview (Anthropic, read October 2026), knowledge cutoffs, prices and the batch discount
- Using the Messages API (Anthropic)
- API pricing (OpenAI, read October 2026)
