Quantization: Smaller Weights, Same Answers (Mostly)
INT8, INT4, and what you actually lose when you shrink a model.
Last time we saw vLLM squeeze thousands of requests into one GPU (graphics processing unit) by tiling the KV (key-value) cache page by page. The weights for those transformer layers still sit in memory as 16-bit floats. This episode gives you the knob that makes a 70B model boot on a single consumer card: quantization.
You can shrink every weight matrix to a quarter or an eighth of its size and get back nearly identical tokens. The catch is not that the model gets stupid. It gets subtle. One poorly handled outlier channel in a single MLP layer can flip a token you’d never think to check.
What does “quantization” actually do to a weight tensor?
Quantization replaces every real number in a tensor with a small integer and a shared scale. The integer is stored in memory. The scale is a single floating-point value that says what one integer leap represents in the original space.
You can think of it like packing books on a shelf where you can only report shelf numbers, not exact positions. You divide the entire row of shelves by a step size, round every book’s location to the nearest shelf, and record the shelf integer together with the step size. A bigger step size saves more space because the integers stay small, but two books that were far apart in different chapters can end up on the same shelf.
In neural networks we use an affine map so the zero of the original space lands exactly on an integer code. For a weight tensor (W) the quantized version (Q) is
[ Q = \text{clip}\left(\text{round}\left(\frac{W}{s}\right) + z,; q_{\min},; q_{\max}\right) ]
where (s) and (z) are the scale and zero-point. To get an approximate real value back you compute (\hat{W} = s(Q - z)). This is the fundamental operation in every quantized linear layer [TensorFlow Lite quantization spec].
Why does per-channel quantization reduce error so much?
A single scale for an entire weight matrix forces every row to share the same step size. If one row has values from -3 to 3 and another from -0.1 to 0.1, the shared scale must cover the wider range. The tiny row then gets squashed into only a few integer bins, losing almost all distinction.
Per-channel quantization solves this by giving each row, or more commonly each output channel, its own scale. The step size adapts. In a weight matrix each column (or row, depending on layout) gets an independent quantizer. The extra cost is storing one scale per channel, which for a 8k-hidden-size Llama-style layer adds a few thousand float32 scalars that are dwarfed by the weight integers.
The difference is dramatic. In a typical comparison, per-tensor quantization of a weight matrix leaves many rows using only a fraction of the integer range; per-channel can reduce quantization error by more than an order of magnitude. That is why every serious deployment toolkit (TensorRT, Qualcomm’s AIMET, PyTorch’s quantize-on-the-fly APIs) defaults to per-channel for weights [NVIDIA TensorRT quantization guide].
- Row 1: 30% of INT8 range
- Row 2: 95%
- Row 3: 15%
- Row 4: 50%
- Row 1: 95%
- Row 2: 95%
- Row 3: 95%
- Row 4: 95%
How does quantized matrix multiplication actually run on the GPU?
The following flow traces what happens inside a single linear layer when both weights and activations are quantized to INT8.
Weights are pre-quantized offline and stored on disk in INT8. At runtime the GPU loads them as 8-bit integers and streams the activation blocks through integer tensor cores. The subtraction of zero-points ensures the affine mapping stays correct, and the 32-bit accumulator captures the product sums exactly.
Symmetric quantization (zero-point = 0 for weights) simplifies the matmul because the weight term disappears. Most toolchains use symmetric schemes for weights and asymmetric for activations, since activations coming out of ReLU or SiLU have a strong positive bias that wastes symmetric ranges [Qualcomm AI Engine quantization guide].
What actually changes in the model’s behavior?
The model does not fail a benchmark. Perplexity climbs a few tenths. For Llama-2-70B quantized with GPTQ to 4-bit, researchers reported a 0.3-point perplexity increase on WikiText-2 while the model shrank from ~140 GB to ~35 GB [GPTQ paper].
The degradation you notice in production is per-token. The quantized model still generates plausible English, still answers most trivia, but one token out of a thousand drifts in a way that matters. A math solution might pick the wrong integer because a softmax probability shifted by 0.0004 after an activation outlier. A JSON (JavaScript Object Notation) field might get a stray comma because the quantizer clipped a large negative value that was crucial for suppressing that token entirely.
This happens because quantization error is not white noise. It is structured by the data distribution. Outlier channels in activations, often concentrated in 0.1% of the feature dimensions, get clipped in INT8 and produce large errors that propagate through the residual stream. LLMs tolerate this for most tokens, but the tail failures are exactly the ones that your users will screenshot.
How do LLM-specific quantization schemes protect against outliers?
Three families dominate:
1. LLM (large language model).int8-style mixed precision keeps the outlier channels in FP16 and quantizes the rest. The authors observed that in models above 6.7B parameters, outlier features occur in systematic, predictable dimensions. They multiply those channels in high precision and everything else in INT8, recovering full FP16 accuracy with about 4× memory savings for the weight matrices [LLM.int8 paper].
2. SmoothQuant shifts the difficulty. It mathematically migrates the per-channel scale of activation outliers into the weight tensors via a diagonal scaling matrix. Activations become easier to quantize, while the weights absorb the larger range. After this transformation, both matrices can use per-tensor INT8 with minimal accuracy loss, yielding true INT8-only inference [SmoothQuant paper].
3. GPTQ and AWQ use post-training rounding optimization. They treat quantization as a layer-wise reconstruction problem: given a calibration set, they find the integer rounding that minimizes the output error of each layer, using second-order information (Hessian) to decide which weights to round up or down. AWQ goes further and scales channels to protect salient weights. Both produce 4-bit models with perplexity only marginally above FP16, while keeping activations in FP16 for speed-sensitive kernels [GPTQ paper].
Where does the serving stack break when you quantize too aggressively?
In the request lifecycle we’ve traced, the first crack usually shows at the softmax bottleneck. With 4-bit weight-only quantization you store less data, but you still compute matmuls in FP16. The speedup is modest unless you also quantize activations. SmoothQuant-style W8A8 unlocks full INT8 matmul, doubling the compute throughput of some kernels.
But full INT8 requires every activation to be quantized just before the matmul. That means the calibration data used to choose activation scales must cover the real distribution of your actual traffic. If your serving traffic includes prompts from a different domain, activation ranges can exceed the calibrated clipping thresholds, silently truncating large values. The model does not crash; it just starts producing low-confidence or repetitive text.
A less obvious failure comes from quantizing the KV cache. When a cache entry is stored as INT8, the precision loss accumulates across attention heads and layers. A 4-bit KV cache can degrade long-context reasoning even when the weight quantization is conservative. This is why production systems often keep the KV cache in FP16 and only shrink the parameters [vLLM documentation on KV cache quantization].
Quick Reference
| Property | Typical value / range |
|---|---|
| Default fp16 weight size | 2 GB per 1B parameters |
| INT8 weight-only size | 1 GB per 1B parameters |
| 4-bit weight-only (GPTQ/AWQ) size | 0.5 GB per 1B parameters |
| Per-channel scheme for weights | One scale per output channel |
| Activation quantization granularity | Per-tensor or token-wise dynamic |
| W8A8 speedup vs FP16 (with integer cores) | ~1.5-2× for large matrix multiplies |
| Overhead from per-channel scales | ~32 bytes per channel (<0.1% of total) |
Frequently Asked Questions
Q: If I quantize a model to INT8 and see 0.2 perplexity increase, does that mean it’s safe for my production task?
Not necessarily. Perplexity is an average over all tokens. A 0.2-point rise can hide a 10% error rate on the tokens that make your JSON parser fail. Run your exact prompts through the quantized model and check token-level agreement with the FP16 baseline, especially for rare tokens or structured outputs.
Q: Why can’t I just quantize everything with per-channel to solve the outlier problem?
Per-channel works well for weights because they are static and their channels have widely different ranges. For activations you do not know the range ahead of time; a per-channel dynamic scheme would need to compute scales for every matrix multiply at runtime, which is expensive. Instead, most systems use per-tensor activation scales and attack outliers with methods like SmoothQuant or mixed precision.
Q: Does quantization affect the attention mechanism differently from MLP layers?
Yes. Attention projections (Q, K, V, O) often have higher sensitivity to quantization because the softmax amplifies small differences. The key and value projections that feed the KV cache are especially problematic since errors accumulate across layers and tokens. In practice, many schemes leave those projections in higher precision even when the MLP is heavily quantized.
Q: How do I know if an accuracy drop is from weight quantization or activation quantization?
Deploy a weight-only variant first. If accuracy is acceptable, then the loss you see in the full W8A8 model comes from activation quantization. That tells you to invest in better calibration data, outlier smoothing (like SmoothQuant), or mixed precision for the activation path without touching the weight quantizer.
Q: Can quantization be combined with speculative decoding or prefix caching?
Absolutely. Quantizing the large model reduces the memory bandwidth needed for verification, so speculative decoding runs even faster. The small draft model is usually already small enough that quantization is not the bottleneck, but you can quantize it too if you want. Prefix caching only caches KV entries; if those are quantized, the same tradeoffs apply.
Test yourself
You maintain a production service that serves a quantized Llama-3-8B model with W8A8 via the vLLM engine. The FP16 model answers a particular query with “The capital of France is Paris.” The quantized model sometimes says “The capital of France is Paris.” and sometimes says “The capital of France is Berlin.” You suspect activation outliers are to blame.
What is the most likely technical mechanism, and how would you confirm it?
Answer: The most likely culprit is an outlier channel in one of the MLP up-projection activations that gets clipped during INT8 quantization, shifting the softmax probabilities for the token that follows “is”. In FP16, a single high-magnitude feature dimension suppresses “Berlin” relative to “Paris” by a tiny but decisive margin. Under INT8 clipping, that suppression weakens, and occasionally “Berlin” wins. To confirm, capture the hidden states for that prompt with both the FP16 and quantized models. Diff the activation tensors per layer, focusing on layers with high 99.9th-percentile magnitudes. If you find a handful of channels that are consistently clipped to the INT8 maximum, apply a targeted fix: either switch those channels to FP16 (LLM.int8 style) or apply SmoothQuant to shift the range into the weights. Then retest the prompt set.
If you want this kind of breakdown every week, how the machinery inside LLM inference really works, not how people talk about it in blog posts, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- TensorFlow Lite quantization spec
- NVIDIA TensorRT quantization guide
- Qualcomm AI Engine quantization guide
- LLM.int8 paper
- SmoothQuant paper
- GPTQ paper
- AWQ paper
- vLLM KV cache quantization documentation