The Economics of Inference: Why API Prices Are What They Are
GPUs, utilization, and margins: the cost structure behind per-token pricing.
Our user’s request has been tokenized, cached, batched, quantized, and accelerated by speculative decoding. The final stop in the serving stack is the billing system. Here the provider attaches a price to the 2,340 input tokens and 512 output tokens that just streamed back. That price looks like a simple number, say $0.004 for the prompt plus $0.015 for the response, but behind it sits a chain of decisions that starts with raw GPU (graphics processing unit) hours and passes through throughput math, architectural efficiency, and deliberate margin targets.
Output tokens cost 4 to 8 times more than input tokens not because the model does more math per token. They cost more because every output token must be generated one after another. That sequential dependence forces the GPU to stay busy longer for exactly the same number of tokens, burning more compute per unit of work. And the list price you pay is often several times higher than the provider’s own hardware cost per token. That gap is not greed. It funds idle capacity, training, and the risk that you might not use the GPU you reserved. Understanding that gap is the key to making rational buy-versus-build decisions.
How do providers convert GPU hours into per-token prices?
GPUs are rented by the hour. An H100 might cost anywhere from roughly $0.60 to $15 per hour depending on the cloud provider, the billing model, and the amount of specialized inference optimization in the stack. Thunder Compute (1)Fireworks AI (14)cloud GPU spot prices (9) To turn that hourly bill into a per-token unit, you need to know how many tokens the GPU can process per second for a given model, precision, and batch size. The formula that most practitioners use is:
Cost per million tokens (CPM) = (GPU price per hour / (tokens per second × 3,600)) × 1,000,000
If a GPU is only 40% utilized, you should divide the hourly price by utilization before running the math. That way effective cost reflects real achieved throughput.
Concrete numbers make this concrete. Suppose you rent an H100 SXM for $2.50 per hour and serve a 70-billion-parameter dense model in FP8. On modern serving stacks you might see roughly 3,000 tokens per second of combined input and output throughput at realistic batch sizes and latency bounds. Plugging those in:
CPM = ($2.50 / (3,000 × 3,600)) × 1,000,000 ≈ $0.23
So raw compute cost is about $0.23 per million tokens. When Google lists Gemini 3.1 Pro output at $12 per million and OpenAI lists GPT-5.4 output at $15 per million, you are looking at a markup of 50× to 65× over that hardware-only floor. The gap is real, and it is there for a reason.
Why are output tokens more expensive than input tokens?
Input tokens are processed in parallel. The entire prompt can be pushed through the transformer layers at once. That keeps the GPU’s compute units packed tightly together. Output tokens must be generated sequentially. The model produces one token, then uses that token to produce the next, paying for the full attention across the growing sequence each time. Because the GPU cannot overlap these decode steps as densely, the time spent per output token is higher, and the cost per token climbs.
Market data across all major providers shows the median output-to-input price ratio is about 4:1. For premium reasoning models, that ratio widens to 6:1 or 8:1. OpenAI pricing (11)Google Gemini pricing (3) The ratio directly mirrors the ratio of decode time to prefill time for a typical chat workload. When you send a 2,000-token prompt and receive an 500-token answer, the prefill step might take 50 ms, but the decode step might take 400 ms. The decoder did eight times the work, and the price tracks that.
How do Mixture-of-Experts architectures change the economics?
In a dense model, every parameter participates in every forward pass. A 200-billion-parameter dense model truly computes 200 billion parameter multiplications per token. A Mixture-of-Experts (MoE (mixture of experts)) model splits the parameters into small expert subnetworks. A router network selects only a few, say 2 out of 64 experts, per token. So a model with 196 billion total parameters might activate only 30 billion per token, an 85% reduction in FLOPs (floating-point operations). Google’s Switch Transformer (16)
That FLOPs drop flows straight into lower GPU time per token. The same H100 can serve many more tokens per second from an MoE model. Providers can therefore charge much less per token while keeping their margins intact. This is why “flash” models built on MoE designs can be priced 20 to 70 times cheaper than dense peers of similar quality on many tasks. The cost difference is real silicon, not marketing.
What do batch and cache discounts actually save the provider?
Batch processing fills the GPU with many requests at once. That improves hardware utilization and amortizes fixed overhead. A provider’s marginal cost per token drops significantly. Passing part of that saving back as a 50% discount for batch jobsGemini batch (3) aligns the price with the reduced cost and nudges traffic into patterns that are easier to serve.
Cached tokens go further. When you cache a long system prompt, the key and value representations are computed once and reused across requests. The provider avoids redoing the attention work for that block. The marginal compute cost for a cached token is a tiny fraction of a fresh token. That is why both Gemini and OpenAI offer cached input tokens at roughly 10% of the normal rate, sometimes with a small storage charge per hour. OpenAI caching (10)Gemini context caching (4) The discount reflects the fraction of the work that remains.
How do providers set the final list price above the hardware cost?
The gap between the compute-only floor and the list price is where the business lives. Providers must amortize the cost of training the model, fund future research, pay for idle capacity during low-traffic periods, and maintain a buffer to absorb usage spikes without degrading latency. A recent industry estimate put the blended hardware cost for Anthropic’s Opus 4.7 on complex agentic tasks at just under $1 per million tokens, while the equivalent list price sits around $5 per million. That implies a gross margin above 70%.
The margin is not free profit. The free tiers these providers run are funded by those margins. A spike in traffic that forces a provider to spin up expensive on-demand capacity to meet latency promises is funded by that margin. And the next generation of training runs, costing hundreds of millions of dollars, is approved on the expectation that inference at scale can provide the return. When you pay $15 per million output tokens, you are buying more than silicon cycles; you are buying a guaranteed capability that has been continuously improved and will continue to be.
Why do long-context windows have a pricing cliff?
Managing a key-value cache for 200,000 tokens is fundamentally different from managing one for 2 million tokens. The memory footprint of the cache grows linearly, but the attention computation across that much context grows quadratically if done naively. Even with optimized attention kernels, the compute and memory pressure increase super-linearly, and the serving system must reserve more GPU memory to accommodate the largest possible prompt a user might send. Providers therefore double the per-token price above 200,000 tokens to reflect the higher cost of serving those extremely long contexts, even if many requests do not use the full window. Gemini Pro pricing (3)
Our user’s prompt was only 2,300 tokens. That fit comfortably inside the base pricing tier. The billing system looks at the length, looks at the model tier, multiplies by the stored rates, adds the output token charge, and sends the amount to the account. No cliff was triggered. But next month, when the product team ships a feature that loads entire codebases into context, that same model call will hit the long-context tier and the price will quietly double.
- Cache memory: ~10 GB
- Attention cost: manageable
- Cache memory: ~100 GB
- Attention cost: 100x higher
The full pipeline from silicon to SKU looks like this:
Quick Reference
| Property | Value |
|---|---|
| Typical H100 cloud price range | $0.57, $14.90/hr |
| Fireworks on-demand H100 (80 GB) | $7/hr |
| L40 price range | $0.56, $1.25/hr |
| Cost model formula | CPM = (GPU $/hr / (tokens/sec × 3600)) × 1e6 |
| Illustrative 70B FP8 CPM at $2.50/hr | ~$0.23 |
| Median output-to-input price ratio | ~4:1 |
| Typical batch discount | 50% off standard pricing |
| Typical cached-token discount | 90% off input pricing |
| Training amortization margin range | 60-80+% |
Frequently Asked Questions
Q: Why are output tokens more expensive than input tokens? Input tokens are processed in parallel, while output tokens must be generated one after another. The sequential decode step keeps the GPU busy longer per token produced, so more time is billed to each output token, raising its price accordingly.
Q: Why do providers offer deep discounts for batch and cached tokens? Batch improves GPU utilization and amortization, reducing the provider’s cost per token. Cached tokens avoid recomputing the key/value states for repeated prompt blocks, which slashes the marginal compute work. The discount passes part of that real saving to you.
Q: How much margin do providers add over their own hardware cost? Estimates for frontier models suggest that list prices can be 50-70× the hardware-only cost, translating to gross margins well above 70% for premium tiers. That margin funds training, idle capacity, and risk absorption.
Q: If I self-host, can I beat API (application programming interface) prices? Yes, if your sustained load keeps GPU utilization consistently high. But you must also cover spike capacity, cold starts, and ongoing engineering. At low or bursty scale, the API’s risk pooling often makes it cheaper than running your own cluster.
Q: Why do long-context prompts trigger a pricing cliff, for example at 200k tokens? Memory and compute for the key-value cache grow super-linearly with prompt length. Providers apply a higher rate beyond a threshold to cover the increased per-request hardware cost and the reserved capacity needed for those large contexts.
Test yourself
Your application currently uses GPT-5.4 at $2.50 per million input tokens and $15 per million output tokens. A typical conversation has 4,000 input tokens and 1,000 output tokens. The batch API offers a 50% discount on both token types but introduces up to 5 seconds of additional latency. Should you switch?
Answer: Switch if half of your user-visible latency budget can absorb the extra 5 seconds and you do not depend on real-time interaction for core value. Without batch, each conversation costs (4,000/1e6 × $2.50) + (1,000/1e6 × $15) = $0.025. With batch, that drops to $0.0125. For 100,000 daily conversations, that saves $1,250 per day. If latency degradation causes even 1% user abandonment and lifetime value per user exceeds several dollars, the math might flip. Measure actual user tolerance with a canary deployment. The discount is real, but it is not free if your product’s stickiness depends on the perception of instant response.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Thunder Compute pricing
- Fireworks AI pricing
- Google Vertex AI Gemini pricing
- Google context caching overview
- OpenAI platform pricing
- OpenAI prompt caching guide
- Run:ai GPU cloud pricing survey
- Google Switch Transformer paper (MoE)
- Gemini batch (3)