Local Inference: Running Models on Your Own Machine

Local inference replaces the model-as-a-service API (application programming interface) call with a computation bound to your own CPU (central processing unit), GPU (graphics processing unit), and memory. The key tools are Ollama and llama.cpp for running models, plus the hardware profile that determines whether a model runs at 5 tokens per second or 50.

Two things decide how well it goes: whether the model and its KV (key-value) cache fit in fast memory, and how fast that memory is. Context length matters more than people expect, because the cache grows with every token. Quantizing the cache, for example keys at 8 bits and values at 4, cuts it by more than half, so more than twice the context fits in the same memory.

Last time we broke down the economics of API inference, finding that a token's price is mostly GPU time and margin. Now we turn to the hardware you own: what it takes to run a model on your own machine, and when "local" is actually the better choice.

Consider the same user request from Part 1, now routed to an Ollama server on a laptop. The prompt is the same, but the whole pipeline (tokenization, prefill, decode, streaming) runs on your machine, not on a remote GPU cluster.

How does local inference differ from API inference in practice?

Local inference changes the trust boundary. The prompt never leaves your machine. No network call, no third-party logs, no data processing agreement. From a compliance perspective, that alone can be the deciding factor for regulated workloads.

It also changes the cost model. Instead of paying per token, you pay once for hardware and then for electricity. A single high-end GPU running a quantized 7B or 8B model can generate millions of tokens per day with no incremental API bill.

The price is that you now manage the memory hierarchy yourself. When you call an API, the provider hides VRAM (the GPU's own memory) pressure, KV cache limits, and model loading. Locally, those become your problem.

The runtime stack is different too. Instead of an HTTP endpoint that hides the model behind a scheduler, you have a program that loads a model file from disk and runs it. llama.cpp is a C/C++ inference engine with command-line tools and its own OpenAI-compatible server, llama-server. Ollama wraps llama.cpp for GGUF models and adds model downloads, management and a REST API; on Apple Silicon, recent versions can also run models on Apple's MLX framework (Ollama's MLX announcement). The abstraction level changes what you can tune.

What hardware configuration actually gives you usable speed?

The first question is capacity. If the model weights and KV cache fit entirely in GPU memory (or in a Mac's unified memory), you get the fastest generation. If part of the model spills to ordinary system RAM, those layers run at system-RAM speed and every token slows down.

A rough rule for GGUF models: Q4_K_M averages about 4.9 bits per weight (llama.cpp's quantize README), so the weights need about 0.6 GB per billion parameters, plus room for the KV cache. Some current examples:

The second question is speed, and it has a simple rule. Each new token requires reading every weight once (Part 1), so for a single user:

decode tokens per second ≈ memory bandwidth ÷ bytes read per token (weights, plus the KV cache at long contexts)

A 7B model at Q4_0 is about 3.8 GB. On an RTX 4090, with 1,008 GB/s of bandwidth, the ceiling is about 1,008 ÷ 3.8 ≈ 265 tokens per second. Real runs land at roughly 60 to 75 percent of the ceiling, as llama.cpp's community benchmarks show (Apple Silicon results and CUDA results, Llama 2 7B, 128 generated tokens, one user):

The same rule explains everything else. The M2 Max runs the 16-bit version of that 7B model at about 25 tokens per second, because there are about three and a half times as many bytes to read. A 70B model at Q4_K_M (about 43 GB) on an M2 Max has a ceiling of about 9 tokens per second. A desktop CPU with common dual-channel DDR5 memory, at roughly 100 GB/s at most, tops out around 20 to 25 tokens per second on a 4 to 5 GB model. A mixture-of-experts model reads only the experts each token uses, so gpt-oss-20b, with 3.6 billion active parameters out of 21 billion (gpt-oss-20b model card), decodes much faster than a dense model of its size, even though all of it must fit in memory.

Hybrid mode (some layers on the GPU, some on the CPU) is a compromise. When a model does not fit in VRAM, Ollama loads part of it into system memory automatically, and ollama ps shows the split, for example "48%/52% CPU/GPU". Layers left on the CPU run from system RAM, with a fraction of the GPU's bandwidth, so each one slows every token; only small activations cross the PCIe bus between the two. Plenty of system RAM matters for hybrid setups with larger models, because the operating system starts swapping to disk if you run short.

Disk speed matters for model loading. GGUF is designed to be memory-mapped, so the OS can page in weights from the file on demand. This lets you start models larger than RAM, but once the model is running, inference speed still depends on the weights fitting in fast memory.

Why does context length blow up VRAM usage?

The KV cache grows linearly with the number of tokens in the sequence. For a transformer with $L$ layers, $H_{\text{KV}}$ key-value heads, head size $d_h$, and a 16-bit KV cache, the total memory is:

bytes = 2 × L × H_kv × d_h × context_length × 2

The first 2 counts keys and values; the last 2 is bytes per 16-bit number. Note that it is the number of KV heads, not query heads: modern models share each KV head across several query heads (grouped-query attention, Part 2). Llama 3.1 8B has 32 layers and 8 KV heads of size 128, so one token costs $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes (128 KiB). That is 1 GiB at 8K tokens, 4 GiB at 32K and 16 GiB at its full 128K. Older models without grouped-query attention, such as Llama 2 7B with 32 KV heads, need four times more per token (512 KiB), or 2 GiB at their full 4K context. Code Llama 34B uses 8 KV heads across 48 layers: 192 KiB per token, about 1.5 GiB at 8K.

This memory is allocated on top of the weights. Even if the weights fit comfortably in VRAM, a long context window can push you over the limit. Many local setups fail here: the model loads fine, but the first request with a long prompt runs out of memory.

The standard fix is KV cache quantization. In llama.cpp you set the key and value types separately, and a quantized value cache needs Flash Attention turned on (llama-server's options):

llama-server -m model.gguf -fa on --cache-type-k q8_0 --cache-type-v q4_0

A q8_0 number takes 8.5 bits and a q4_0 number 4.5 bits, counting their shared scales, against 16 for f16. Keys at q8_0 and values at q4_0 average 6.5 bits, about 59% less memory than f16, so about 2.5 times the context fits in the same space. Ollama sets one type for both keys and values: with Flash Attention enabled (Ollama turns it on automatically where the hardware supports it, or you can set OLLAMA_FLASH_ATTENTION=1), set OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache or q4_0 to quarter it (Ollama FAQ). The split keys-at-8, values-at-4 setting needs llama.cpp directly.

Why give keys more bits than values? Keys set the attention scores, which decide how much each earlier token counts, while values carry the information that gets mixed. The KIVI paper found that the two have different statistics: the key cache quantizes best per channel and the value cache per token (KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache). A community benchmark, KVSplit (May 2025, TinyLlama at 8K context on an M4 MacBook Pro), shows the effect at the same memory cost: 8-bit keys with 4-bit values raised perplexity by 0.86%, while 4-bit keys with 8-bit values raised it by 6.06%, both with the cache 59% smaller. That is one small model on one machine, so test on your own model before relying on it.

This asymmetry means you can more than double the context that fits on the same machine by keeping keys at higher precision and compressing values harder. For local inference, it is one of the most useful optimizations you can make.

How do llama.cpp and Ollama load and execute a model?

llama.cpp loads a GGUF file and executes the transformer graph on CPU, GPU, or a mix of both, with backends that include CUDA, Metal, Vulkan, HIP, SYCL and plain CPU. The GGUF format stores weights, tokenizer metadata, and hyperparameters in a single file that can be memory-mapped (GGUF specification).

When you run ollama run llama3.2, Ollama starts a server that downloads the model (if it is not cached) and loads it with llama.cpp. It keeps the model in memory for five minutes after the last request by default, so later requests skip the load, and it listens on port 11434. Your request hits that endpoint, the prompt is tokenized, and prefill begins.

The internal flow looks like this:

During prefill, the entire prompt is processed in parallel. The graph planner in llama.cpp allocates tensor nodes and assigns them to the best available backend. Large matrix multiplications go to the GPU if present.

Decode is a loop. At each step, the model takes the last generated token, pushes it through every layer, and attends over the cached keys and values for all previous positions. Each step must read every weight once, which is why the bandwidth rule above predicts speed so well; at long contexts, reading the KV cache adds to the bill.

Ollama adds a layer on top. It manages the model lifecycle, exposes a REST API, and handles streaming. It also provides a CLI (command-line interface) for pulling models, setting parameters, and checking logs.

What does a production-grade local server look like?

When several people share one local server, concurrency matters. Ollama can process a few requests in parallel per model (OLLAMA_NUM_PARALLEL; the default is one), and llama-server has continuous batching and multiple parallel slots built in. For dozens of users, a dedicated serving engine such as vLLM batches far more efficiently, with the paged KV cache and continuous batching that Part 3, Batching and PagedAttention: How vLLM Serves Hundreds at Once, explains.

TensorRT-LLM, NVIDIA's serving library, now runs on PyTorch with NVIDIA-tuned kernels; its older TensorRT engine backend has been removed (TensorRT backend removal notice). It uses CUDA Graphs to cut the CPU overhead of launching GPU kernels, and an overlap scheduler that hides CPU work behind GPU computation (TensorRT-LLM architecture overview).

For a local team running a private RAG (retrieval-augmented generation) system, one of these serving engines can handle many simultaneous requests on a single workstation GPU. The setup is more complex than Ollama, but the total throughput is much higher.

Quick Reference

Frequently Asked Questions

Q: Can I run a 70B model on a laptop with 64 GB of RAM? Yes, on the right laptop. A 64 GB Apple Silicon Mac can run a 70B model at Q4_K_M (about 43 GB) on its GPU through Metal; on an M2 Max the bandwidth rule gives a ceiling of about 9 tokens per second, so expect around 5 to 6 in practice. A 64 GB PC without a big GPU has to run it from system RAM, and at about 100 GB/s you get only 1 to 2 tokens per second. Fine for batch summarization, not for chat.

Q: Does local inference completely eliminate privacy risk? It eliminates the risk of data leaving your machine. The prompt and generated tokens never hit a third-party server. But you still need to secure the local system: disk encryption, access controls, and proper disposal of model files.

Q: How do I choose between llama.cpp and Ollama? Both have a command-line interface and an HTTP API, and for GGUF models Ollama runs llama.cpp underneath. Ollama adds one-command model downloads and management; llama.cpp exposes every tuning flag, such as separate key and value cache types. Start with Ollama, and move to llama.cpp when you need a setting Ollama does not offer.

Q: Why does VRAM limit context length even if the model fits? Because the KV cache grows linearly with context length and sits in VRAM alongside the weights. Llama 3.1 8B at Q4_K_M takes about 4.9 GB, and 32K tokens of context add another 4 GiB. That can push the total over an 8 GB GPU's limit.

Q: What's the fastest local inference setup I can assemble? For one user, the fastest setup is the one with the most memory bandwidth that holds your model: in llama.cpp's benchmarks an RTX 5090 runs a 7B model at Q4_0 at about 290 to 300 tokens per second. For many users at once, a serving engine with continuous batching gets far more total tokens per second out of the same card.

Test yourself

You want to serve Code Llama 34B at Q4_K_M (about 20 GB of weights) on a single GPU with 16 GB of VRAM, with 8K tokens of context. Can you run this setup? If not, what changes would make it work?

Answer: Not fully on the GPU: 20 GB of weights cannot fit in 16 GB, and 8K tokens of KV cache add about 1.5 GiB more ($2 \times 48 \times 8 \times 128 \times 2$ bytes = 192 KiB per token). You have three ways out. Hybrid offload: keep about 14 GB of layers plus the cache on the GPU and leave about 6 GB of layers in system RAM. Those CPU layers set the pace: at roughly 100 GB/s they alone take about 60 ms per token, so expect around 10 tokens per second at best. A smaller model: a 14B model at Q4_K_M is about 9 GB and fits with plenty of room for 8K of context, at several times the speed. A harsher quantization: the 34B at Q2_K is about 14 GB, which leaves almost no room for the cache and costs noticeable quality. For most uses the 14B model is the better trade; if you need the 34B, a 24 GB card is the clean fix.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• llama.cpp on GitHub
• llama-server README (llama.cpp)
• llama.cpp quantize README
• Performance of llama.cpp on Apple Silicon (llama.cpp discussion 4167)
• Performance of llama.cpp with CUDA (llama.cpp discussion 15013)
• GGUF format specification
• Ollama FAQ
• Ollama is now powered by MLX on Apple Silicon in preview (Ollama, March 2026)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al. 2024)
• KVSplit (community benchmark, May 2025)
• gpt-oss-20b model card (OpenAI on Hugging Face)
• TensorRT-LLM architecture overview (NVIDIA)
• TensorRT backend removal notice (TensorRT-LLM, October 2026)
• GeForce RTX 50 series compared (NVIDIA)