The Economics of Inference: Why API Prices Are What They Are

Our user's request has been tokenized, cached, batched, quantized, and accelerated by speculative decoding. The final stop in the serving stack is the billing system. Here the provider attaches a price to the 2,340 input tokens and 512 output tokens that just streamed back. At GPT-5.4's list prices (USD as of October 2026: $2.50 per million input tokens and $15 per million output tokens), that comes to about $0.006 for the prompt plus $0.008 for the response. Behind that simple number sits a chain of decisions that starts with raw GPU (graphics processing unit) hours and passes through throughput math, architectural efficiency, and deliberate margin targets.

Output tokens cost about 5 to 6 times more than input tokens at today's list prices, and not because the model does more math per token. They cost more because every output token must be generated one after another. Each decode step waits on memory rather than doing math (Part 1), so an output token occupies far more GPU-seconds than a prompt token. And the list price you pay is usually several times higher than the provider's own hardware cost per token. That gap is not greed. It funds idle capacity, training, and the risk that you might not use the GPUs reserved for you. Understanding that gap is the key to making rational buy-versus-build decisions.

How do providers convert GPU hours into per-token prices?

GPUs are rented by the hour. As of October 2026, an H100 rents for roughly $2 to $4 per GPU-hour on GPU clouds: RunPod's pricing lists the H100 SXM at $2.69 to $3.99 depending on the tier, and Thunder Compute's pricing shows $3.20. Managed inference platforms charge more for a dedicated GPU, such as $8 per hour on Fireworks AI's pricing page. To turn an hourly bill into a per-token unit, you need to know how many tokens the GPU can process per second for a given model, precision, and batch size. The formula that most practitioners use is:

Cost per million tokens (CPM) = (GPU price per hour / (tokens per second × 3,600)) × 1,000,000

If a GPU is only 40% utilized, divide the hourly price by the utilization (0.4) before running the math. That way the effective cost reflects real achieved throughput.

Concrete numbers make this clearer; the ones below are illustrative. Suppose you rent two H100 SXM GPUs at $3 per hour each and serve a 70-billion-parameter dense model in FP8 across them. A single 80 GB card would hold the 70 GB of FP8 weights but leave little room for the KV cache, which keeps batches small (Part 2); two cards leave room for real batches. Say your traffic, mostly prompt tokens, averages 6,000 tokens per second across the pair. Plugging those in:

CPM = ($6 / (6,000 × 3,600)) × 1,000,000 ≈ $0.28

So raw compute cost is roughly $0.28 per million tokens in this example. The fair comparison is a hosted open 70B model, because anyone can serve the same weights. Hosted Llama 3.3 70B lists from $0.10 per million input tokens and $0.32 per million output tokens for the Turbo version on DeepInfra's pricing page to $1.04 for either on Together AI's pricing page (USD, October 2026). That is close to this kind of floor, because many providers compete to serve the same open model. Frontier models are much larger and their serving costs are not public, so dividing their list prices by a 70B floor would say nothing useful.

Why are output tokens more expensive than input tokens?

Input tokens are processed in parallel. The entire prompt can be pushed through the transformer layers at once, which keeps the GPU's compute units busy. Output tokens must be generated sequentially. The model produces one token, then uses that token to produce the next, reading all of its weights and the growing KV cache each time. Because each decode step does little math per byte read, the GPU time spent per output token is much higher.

An illustrative example shows the size of the gap. With a 2,000-token prompt and a 500-token answer, prefill might take about 0.2 seconds, while decode takes 5 to 10 seconds at 50 to 100 tokens per second. For that single user, an output token costs the GPU around a hundred times more time than a prompt token. Batching many users into each decode step shares the weight reads across all of them and narrows that gap a lot, and competition sets the rest. That's why list prices land at a ratio of about 5 to 6: the standard rates on OpenAI's API pricing, Google's Gemini API pricing and Anthropic's pricing give the ratios in the chart.

How do Mixture-of-Experts architectures change the economics?

In a dense model, every parameter participates in every forward pass: a 70-billion-parameter dense model does about 70 billion multiply-adds per token. A Mixture-of-Experts (MoE) model splits its feed-forward layers into many expert subnetworks, and a small router network picks a few experts for each token. The Switch Transformer, an early example, sends each token to a single expert (Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity). DeepSeek-V3 has 671 billion parameters in total but activates only 37 billion for each token (DeepSeek-V3 Technical Report), so each token costs roughly the math of a 37B dense model.

That cut in math flows straight into lower GPU time per token wherever the GPU is compute-bound, as in prefill and large batches, and at small batches each token also reads fewer weights, since only the chosen experts are touched. What MoE does not cut is memory capacity: all 671 billion parameters must still sit in GPU memory, so the model needs as many GPUs as a dense model of its total size. Providers can price MoE models lower per token, but how much lower depends on traffic and batch sizes, and most providers do not say which of their models are MoE.

What do batch and cache discounts actually save the provider?

Batch processing lets the provider fill its GPUs with work that can wait. Requests that do not need an answer now can run in large, efficient batches and in hours when GPUs would otherwise sit idle, which lowers the provider's cost per token. OpenAI, Anthropic and Google all pass part of that saving back as a 50% discount, in exchange for results that arrive within 24 hours instead of seconds (OpenAI's Batch API guide, Anthropic's batch processing docs, Gemini's Batch Mode docs).

Cached tokens go further. When you cache a long system prompt, its keys and values are computed once and reused across requests, so the provider skips the prefill work for that block. The marginal compute cost for a cached token is a small fraction of a fresh one. That's why cached input usually costs about 10% of the normal input rate: $0.25 instead of $2.50 per million on GPT-5.4, $0.50 instead of $5 for cache reads on Claude Opus 4.7, and $0.20 instead of $2.00 on Gemini 3.1 Pro, which also charges $4.50 per million tokens per hour of storage (USD, October 2026). Some newer models discount cache reads even further. The discount reflects the fraction of the work that remains.

How do providers set the final list price above the hardware cost?

The gap between the compute-only floor and the list price is where the business lives. Providers must amortize the cost of training the model, fund future research, pay for idle capacity during low-traffic periods, and keep a buffer to absorb usage spikes without degrading latency.

Real margins are not published, but analysts estimate them. SemiAnalysis estimated in May 2026 that gross margins on Anthropic's inference infrastructure had risen from 38% to over 70% (AI Value Capture: The Shift To Model Labs). A gross margin of 70% means the price is about 3.3 times the serving cost ($1 \div 0.3$); a 50-times markup would mean a 98% margin. The same analysis shows why list prices can mislead: it put the effective blended price of agentic work on Opus 4.7 at about $0.99 per million tokens, against list prices of $5 for input and $25 for output, because such workloads send long prompts that are mostly cheap cache hits.

The margin is not free profit. The free tiers these providers run are funded by it. A traffic spike that forces a provider to spin up expensive on-demand capacity to meet latency promises is funded by it. And the next generation of training runs, costing hundreds of millions of dollars, is approved on the expectation that inference at scale can provide the return.

Why do long-context windows have a pricing cliff?

Managing a key-value cache for 200,000 tokens is a different job from managing one for 2 million tokens. The memory footprint of the cache grows linearly with length, but attention work over the whole prompt grows with its square if done naively. Even with optimized attention kernels, long prompts tie up a lot of GPU memory for a long time, and the serving system must be able to fit the largest prompt a user might send.

Some providers therefore charge more per token above a threshold. Gemini 3.1 Pro doubles its input price ($2 to $4 per million) and raises its output price 1.5 times ($12 to $18) for prompts over 200,000 tokens; GPT-5.4 does the same above 272,000 tokens ($2.50 to $5 for input, $15 to $22.50 for output). Others don't: Anthropic bills Claude 4.6 and later models (except Claude Haiku 5.5) at standard rates across the full 1M-token window (all USD, October 2026).

Our user's prompt was only 2,340 tokens. That fit comfortably inside the base pricing tier. The billing system looks at the length, looks at the model tier, multiplies by the stored rates, adds the output token charge, and sends the amount to the account. No cliff was triggered. But next month, when the product team ships a feature that loads entire codebases into context, the same call on GPT-5.4 or Gemini 3.1 Pro will cross the threshold: the input price doubles and the output price rises by half.

The full pipeline from silicon to SKU looks like this:

Quick Reference

Frequently Asked Questions

Q: Why are output tokens more expensive than input tokens? Input tokens are processed in parallel, while output tokens must be generated one after another. Each decode step waits on memory, so it keeps the GPU busy longer per token produced, and more GPU time is billed to each output token.

Q: Why do providers offer deep discounts for batch and cached tokens? Batch work can wait, so it fills idle GPU time in large, efficient batches, reducing the provider's cost per token. Cached tokens skip the prefill work for repeated prompt blocks, which slashes the marginal compute. The discount passes part of that real saving to you.

Q: How much margin do providers add over their own hardware cost? Nobody publishes it, but analysts estimate it. SemiAnalysis estimated that gross margins on Anthropic's inference infrastructure rose from 38% to over 70% by May 2026. Those margins mean list prices of about 1.6 times serving cost at the low end and about 3.3 times or more at the high end, far from the 50 times sometimes claimed. That margin funds training, idle capacity, and risk absorption.

Q: If I self-host, can I beat API (application programming interface) prices? Yes, if your sustained load keeps GPU utilization consistently high. But you must also cover spike capacity, cold starts, and ongoing engineering. At low or bursty scale, the API's risk pooling often makes it cheaper than running your own cluster.

Q: Why do long-context prompts trigger a pricing cliff, for example at 200k tokens? The KV cache memory grows linearly with prompt length, and attention compute grows faster than linearly. Long prompts also tie up memory for longer. Some providers apply a higher rate beyond a threshold to cover that; others, such as Anthropic for its recent Claude models, charge the same rate across the whole window.

Test yourself

Your application uses GPT-5.4 at $2.50 per million input tokens and $15 per million output tokens. A typical conversation has 4,000 input tokens and 1,000 output tokens, and you run 100,000 conversations a day. The Batch API halves both prices but returns results within 24 hours. Should you move your live chat to it? If not, what would you do instead?

Answer: Each conversation costs 4,000 × $2.50 / 1,000,000 + 1,000 × $15 / 1,000,000 = $0.01 + $0.015 = $0.025, or $2,500 a day. Batch would halve that to $0.0125 and save $1,250 a day, but answers that arrive hours later make no sense for live chat. Use Batch for offline work instead, such as evaluations, bulk summaries or nightly tagging. For the chat itself, look at prompt caching first: if 3,000 of the 4,000 input tokens are a shared system prompt, those cost $0.25 per million instead of $2.50, and a conversation drops to about $0.018, roughly 27% less. Requests that can wait a little but not a day can use OpenAI's Flex processing, billed at Batch rates in exchange for slower responses and occasional unavailability. A smaller, cheaper model for simple turns is the other lever.

Next: Local Inference: Running Models on Your Own Machine. It takes the same ideas, memory, bandwidth and quantization, onto your own GPU or laptop.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• API pricing (OpenAI)
• Batch API guide (OpenAI)
• Flex processing guide (OpenAI)
• Gemini API pricing (Google)
• Batch Mode (Gemini API documentation)
• Pricing (Anthropic)
• Batch processing (Anthropic)
• RunPod pricing
• Thunder Compute pricing
• Fireworks AI pricing
• DeepInfra pricing
• Together AI pricing
• AI Value Capture: The Shift To Model Labs (SemiAnalysis, May 2026)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus, Zoph and Shazeer 2021)
• DeepSeek-V3 Technical Report (DeepSeek-AI 2024)