Local Inference: Running Models on Your Own Machine
Ollama, llama.cpp, and what hardware actually gets you usable speed.
Local inference replaces the model-as-a-service API (application programming interface) call with a computation bound to your own CPU (central processing unit), GPU (graphics processing unit), and memory. The key systems are Ollama and llama.cpp for running models, plus the hardware profile that determines whether a model runs at 5 tokens per second or 50.
But the real surprise is that context length, not model size, often kills local performance. A single optimization, keeping KV (key-value) cache keys at higher precision than values, can double your usable context window with almost no quality loss.
Last time we broke down the economics of API inference, finding that a token’s cost is mostly GPU utilization and margin. Now we turn to the hardware you own: what it takes to run a model on your own laptop, and when “local” is actually faster than a remote API.
Consider the same user request from Part 1, now routed to an Ollama server on a MacBook Pro. The prompt is the same, the model is the same size, but the entire token pipeline, tokenization, prefill, decode, streaming, runs on your machine, not on a remote GPU cluster.
How does local inference differ from API inference in practice?
Local inference changes the trust boundary. The prompt never leaves your machine. No network call, no third-party logs, no data processing agreement. From a compliance perspective, that alone can be the deciding factor for regulated workloads.
It also changes the cost model. Instead of paying per token, you pay once for hardware and then for electricity. A single high-end GPU like an RTX 4090, running a quantized 7B model, can generate millions of tokens per day with no incremental API bill.
The price is that you now manage the memory hierarchy yourself. When you call an API, the provider abstracts away VRAM (video random-access memory) pressure, KV cache eviction, and model loading. Locally, those become your problem.
The runtime stack is fundamentally different. Instead of an HTTP endpoint that hides the model behind a scheduler, you have a binary that loads a model file from disk and runs it. Ollama wraps llama.cpp with a REST API. llama.cpp is a pure C++ inference engine. Both rely on the same underlying compute, but the abstraction level changes what you can tune.
What hardware configuration actually gives you usable speed?
The single most important resource is VRAM. If the model weights and KV cache fit entirely in GPU memory, you get the fastest generation. If they spill to system RAM, speed drops by an order of magnitude.
A rough rule of thumb for GGUF models: a Q4_K_M quantization needs about 0.6-0.7 GB of VRAM per billion parameters. So a 7B model fits in a 6 GB GPU; a 34B model needs about 20 GB, which puts it in RTX 4090 or A100 territory.
CPU-only inference is viable but slow. On a modern desktop CPU with AVX2, a 7B model at Q4_K_M might reach 10-15 tokens per second. On an M2 Max with Metal acceleration, it can hit 20-25. That’s usable for batch processing or prototyping, but not for interactive chat where you want 40+ tokens per second.
Hybrid mode, some layers on GPU, some on CPU, is a compromise. ollama automatically offloads as many layers as VRAM allows. Each layer that stays on the CPU adds latency; the PCIe bus becomes a bottleneck. 32 GB of system RAM is a practical minimum for hybrid setups with larger models, because the OS ends up paging between RAM and swap if you’re short.
Disk speed matters for model loading. GGUF is designed to be memory-mapped, so the OS can page in weights from the file on demand. This lets you run models larger than RAM, but once the model is loaded, actual inference speed still depends on the working set fitting in faster memory.
A quick reference for common hardware profiles:
| Model size | Quantization | VRAM needed (approx) | Speed on RTX 4090 | Speed on M2 Max (CPU+GPU) |
|---|---|---|---|---|
| 7B | Q4_K_M | 5.5 GB | 80+ tok/s | 20-25 tok/s |
| 34B | Q4_K_M | 20 GB | 35-45 tok/s | 7-10 tok/s |
| 70B | Q2_K | 24 GB | 20-25 tok/s | 4-6 tok/s |
| 70B | Q4_K_M | 43 GB | requires 2×GPU | 2-3 tok/s |
Why does context length blow up VRAM usage?
The KV cache grows linearly with the number of tokens in the sequence. For a transformer with L layers, H heads, head dimension d, and FP16 KV cache, the total memory is:
bytes = 2 × L × H × d × context_length × 2
That extra factor of 2 at the end is for the 16-bit data type. For a 7B model with 32 layers, 32 heads, and 128-dim heads, an 8K context adds about 2.1 GB of KV cache. For a 34B model, it can add 8 GB or more.
This memory is allocated on top of the weight storage. Even if weights fit comfortably in VRAM, a long context window can push you over the limit. Many local setups fail here: the model loads fine, but the first request with a long prompt triggers an out-of-memory error.
The standard fix is KV cache quantization. llama.cpp lets you pass --cache-type-k q8_0 --cache-type-v q4_0 to store keys at 8-bit and values at 4-bit. This cuts KV cache size by roughly 50% with minimal quality loss.
But the real trick is asymmetric quantization. Research on KVSplit shows that keys need higher precision than values. Keys determine the attention pattern, the scores that decide which tokens attend to which. Errors in keys distort the entire attention map. Values, once the attention is computed, affect only the information passed forward, and errors there are more localized.
A configuration like K8V4 (8-bit keys, 4-bit values) achieves about 59% KV memory reduction with only 0.86% perplexity loss. K4V8 (4-bit keys, 8-bit values) gives the same memory reduction but a 6.06% perplexity loss. That’s a seven-fold quality difference for the same bit budget. KVSplit PR (pull request)
This asymmetry means you can double your effective context window on the same MacBook by keeping keys at higher precision and compressing values aggressively. For local inference, it’s one of the highest-leverage optimizations you can make.
How do llama.cpp and Ollama load and execute a model?
llama.cpp is a C++ library that loads a GGUF file and executes the transformer graph on CPU, GPU, or a mix of both. The GGUF format stores weights, tokenizer metadata, and hyperparameters in a single file that can be memory-mapped. GGUF spec
When you run ollama run llama3.2, Ollama starts a server that downloads the model (if not cached) and passes it to llama.cpp. The server then spins up an HTTP endpoint. Your request hits that endpoint, the prompt is tokenized, and llama_decode is called for the prefill phase.
The internal flow looks like this:
During prefill, the entire prompt is processed in parallel. The graph planner in llama.cpp allocates tensor nodes and assigns them to the best available backend (CUDA, Metal, CPU). Large matrix multiplications go to the GPU if present.
Decode is a loop. At each step, the model takes the last generated token, computes its query vector, and uses the cached keys and values for all previous positions. This is a memory-bound operation: the bottleneck is fetching KV cache entries, not raw arithmetic.
Ollama adds a thin layer on top. It manages model lifecycle, exposes a REST API, and handles streaming. It also provides a CLI (command-line interface) for pulling models, setting parameters, and checking logs. Under the hood, it’s still llama.cpp doing the heavy lifting.
What does a production-grade local server look like?
When you need to serve multiple concurrent users locally, Ollama’s single-request model starts to show its limits. That’s where vLLM and TensorRT-LLM come in.
vLLM introduced paged attention. Instead of storing KV cache as a contiguous array per sequence, it breaks the cache into fixed-size blocks and uses a page table to map logical token positions to physical blocks. This lets the scheduler reclaim blocks when a request finishes, pack sequences tightly, and support long contexts even with fragmented memory. vLLM paper
- Pre-allocated per sequence
- Memory fragmentation
- Wasted capacity when idle
- Fixed-size blocks
- Page table maps logical to physical
- Dynamic allocation, no waste
Continuous batching, shared by vLLM and Hugging Face TGI, keeps the GPU busy. As soon as one request finishes, another request’s decode step can be batched in. The scheduler interleaves prefill and decode phases across requests, so the GPU never idles waiting for the next token.
TensorRT-LLM goes further. It compiles the entire model graph into a TensorRT engine, fusing operations and selecting the best kernels for the specific GPU. It uses CUDA Graphs to capture the decode step for a fixed batch size, reducing CPU launch overhead by up to 22%. And it can overlap CPU work for the next step with the current GPU computation, hiding latency. TensorRT-LLM repo
For a local team running a private RAG (retrieval-augmented generation) system, this stack can serve dozens of simultaneous requests on a single A100 or RTX 4090. The setup is more complex than Ollama, but the throughput is an order of magnitude higher.
Quick Reference
| Property | Value |
|---|---|
| Default GGUF quantization for 7B | Q4_K_M |
| Typical VRAM per billion params (Q4) | ~0.65 GB |
| KV cache memory per token (FP16, 7B model) | ~0.26 MB |
| KV cache memory cut with K8V4 | ~59% reduction |
| llama.cpp backend options | CPU, CUDA, Metal, Vulkan, SYCL |
| Ollama default port | 11434 |
| vLLM paged attention block size | 16 tokens (typical) |
Frequently Asked Questions
Q: Can I run a 70B model on a laptop with 64 GB of RAM? Yes, if you use a heavily quantized GGUF (Q2_K) and CPU-only inference. You’ll get around 2-4 tokens per second on an M2 Max. It’s not interactive, but it works for batch summarization.
Q: Does local inference completely eliminate privacy risk? It eliminates the risk of data leaving your machine. The prompt and generated tokens never hit a third-party server. But you still need to secure the local system: disk encryption, access controls, and proper disposal of model files.
Q: How do I choose between llama.cpp and Ollama? If you want a CLI and fine-grained control over backends, use llama.cpp directly. If you need a REST API, model management, and a simpler experience, Ollama is the right choice. Both use the same underlying engine.
Q: Why does VRAM limit context length even if the model fits? Because the KV cache grows linearly with context length and is stored in VRAM alongside the weights. A 7B model at Q4 fits in 6 GB, but an 8K context can add another 2 GB. That can push the total over your GPU’s limit.
Q: What’s the fastest local inference setup I can assemble? A high-end GPU with TensorRT-LLM, 4-bit weight quantization, and a paged KV cache. On an RTX 4090, you can push a 7B model above 100 tokens per second with continuous batching.
Test yourself
You have a 34B model at Q4_K_M, a single GPU with 16 GB VRAM, and you need to serve requests with 8K context. The model weights consume 20 GB, which already exceeds your VRAM. Can you run this setup? If not, what changes would make it work?
Answer: No, the weights alone exceed the 16 GB VRAM by 4 GB. You cannot load the full model on the GPU. Even if you use hybrid execution, offloading only half the layers to GPU, the remaining layers on CPU will slow everything down. And the KV cache for 8K context will add several GB more, making the VRAM pressure worse. To make it work, you need a GPU with at least 24 GB VRAM, or you must drop to a more aggressive quantization (Q2_K) to shrink the weight footprint below 16 GB. Alternatively, you could reduce the context length to 2K, which cuts the KV cache size significantly, but that defeats the purpose of 8K. In practice, a 34B model at Q4_K_M requires a GPU with 24 GB or more, such as an RTX 4090 or A10.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- llama.cpp GitHub
- GGUF format specification
- Ollama GitHub
- vLLM: PagedAttention paper
- TensorRT-LLM GitHub
- KVSplit asymmetric quantization PR