Quantization: Smaller Weights, Same Answers (Mostly)
INT8, INT4, and what you actually lose when you shrink a model.
Last time we saw vLLM fit hundreds of requests onto one GPU (graphics processing unit) by storing the KV (key-value) cache in small blocks. The weights for those transformer layers still sit in memory as 16-bit numbers. This part gives you the knob that lets a 70B model fit on one 80 GB datacenter GPU, or an 8B model run on a laptop: quantization.
You can shrink every weight matrix to half (INT8) or a quarter (INT4) of its 16-bit size and get back nearly identical tokens. The catch is not that the model gets stupid. It gets subtle. One poorly handled outlier channel in a single MLP layer (the feed-forward block inside every transformer layer) can flip a token you'd never think to check.
What does "quantization" actually do to a weight tensor?
Quantization replaces every real number in a tensor with a small integer and a shared scale. The integer is stored in memory. The scale is a single floating-point value that says what one integer step represents in the original space.
You can think of it like packing books on a shelf where you can only report shelf numbers, not exact positions. You divide the entire row of shelves by a step size, round every book's location to the nearest shelf, and record the shelf integer together with the step size. A bigger step size saves more space because the integers stay small, but two books that sat close together get recorded on the same shelf, so you can no longer tell them apart.
In neural networks we use an affine map so the zero of the original space lands exactly on an integer code. For a weight tensor the quantized version is
where and are the scale and zero-point. To get an approximate real value back you compute . This is the same mapping the LiteRT 8-bit quantization specification writes as real value = (int8 value minus zero point) × scale, and it is the basic operation in every quantized linear layer.
Here is one worked by hand. Take four weights, , and quantize them to symmetric INT8 (zero-point 0, integers from to ). The scale maps the largest magnitude to 127:
Dividing each weight by and rounding gives the integers (for example , which rounds to 12). Multiplying back by returns : every weight within half a step of where it started.
Now add one outlier, 13.0, to the same group. The scale becomes , ten times coarser. The integers are now , and they come back as . The 0.04 has vanished to zero and the 0.12 has lost 15 percent of its value, all because one large number stretched the scale. That single effect explains most of what follows.
Why does per-channel quantization reduce error so much?
A single scale for an entire weight matrix forces every row to share the same step size. If one row has values from to and another from to , the shared scale must cover the wider range. The tiny row then gets squashed into only a few integer bins, losing almost all distinction, exactly like the 0.04 above.
Per-channel quantization solves this by giving each row, or more commonly each output channel, its own scale. The step size adapts. The extra cost is storing one scale per channel: for an 8,192 by 8,192 matrix that is 8,192 scales against 67 million weights, which is tiny.
The gain is large whenever rows differ in range, which in real weight matrices they do. NVIDIA's TensorRT guide, for example, recommends per-channel quantization for weights and per-tensor quantization for activations (Working with Quantized Types).
- Row 1: 30% of INT8 range
- Row 2: 100%
- Row 3: 15%
- Row 4: 50%
- Row 1: 100%
- Row 2: 100%
- Row 3: 100%
- Row 4: 100%
At 4 bits there are only 16 integer levels, so even one scale per row is too coarse. 4-bit methods use group-wise scales instead: one scale (and sometimes a zero-point) for every 128 weights. GPTQ and AWQ both use groups of 128, and the GPTQ paper puts the overhead at about 0.15 extra bits per weight. One 16-bit scale per 128 weights is bits per weight, about 3 percent of a 4-bit model; with a zero-point it comes to roughly 0.15 to 0.25 bits.
How does quantized matrix multiplication actually run on the GPU?
The following flow traces what happens inside a single linear layer when both weights and activations are quantized to INT8.
Weights are quantized offline and stored on disk in INT8. At runtime the GPU loads them as 8-bit integers and streams the activation blocks through its integer tensor cores (the units built for fast matrix math). The subtraction of zero-points keeps the affine mapping correct, and the 32-bit accumulator captures the product sums exactly.
Symmetric quantization (zero-point = 0) simplifies the matmul because the zero-point terms disappear. The LiteRT spec makes weights symmetric and activations asymmetric, since activations coming out of ReLU or SiLU lean strongly positive and would waste half of a symmetric range. TensorRT uses symmetric scales for both.
What actually changes in the model's behavior?
The main quality number in this field is perplexity: roughly, how surprised the model is by real text it has not seen, where lower is better. A rise of a few tenths is small; a rise of several points is a damaged model.
Good 4-bit methods barely move it. The AWQ paper's table for Llama 2 70B on WikiText-2 shows perplexity going from 3.32 in FP16 to 3.42 with GPTQ and 3.41 with AWQ at 4 bits (groups of 128), and to 3.88 and 3.74 at 3 bits (AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration). Meanwhile the weights shrink from about 140 GB to about 36 to 37 GB, counting the group scales.
The degradation you notice in production is per-token. The quantized model still generates plausible English and still answers most trivia, but now and then a token drifts in a way that matters. For example, a math solution might pick the wrong integer because a probability shifted slightly after an activation outlier, or a JSON (JavaScript Object Notation) field might get a stray comma because a large negative value that suppressed that token lost precision.
This happens because quantization error is not white noise. It is structured by the data. In larger models a few activation dimensions carry huge values; the LLM.int8() authors found them in only about 0.1% of feature dimensions. With a scale set by the largest value, such an outlier stretches the scale and crushes every other value in its tensor into a few integer steps, as in the worked example; with a fixed, calibrated range, it gets clipped instead. Either way the error is large, and it travels through the residual stream (the running sum that carries each layer's output to the next). LLMs tolerate this for most tokens, but the tail failures are exactly the ones your users will screenshot.
How do LLM-specific quantization schemes protect against outliers?
Three families dominate:
1. LLM.int8()-style mixed precision keeps the outlier dimensions in FP16 and quantizes the rest. The authors found that at around 6.7B parameters outlier features appear suddenly, in a few systematic dimensions. Multiplying those in 16-bit lets more than 99.9% of values be multiplied in 8-bit with no loss of accuracy, while the memory needed for inference is cut in half: 1.96 times smaller for BLOOM-176B (LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale). The donut below shows that split, from the same paper.
2. SmoothQuant shifts the difficulty. It moves the hard part of quantization from the activations to the weights with a mathematically equivalent transformation: it divides each activation channel by a factor and multiplies the matching weight row by the same factor. Activations become easier to quantize, while the weights absorb the larger range. After this, both can use simple INT8 scales, giving W8A8 inference (8-bit weights and 8-bit activations) for all the large matrix multiplies, while light operations such as softmax and LayerNorm stay in FP16. The paper reports up to 1.56 times speedup and 2 times less memory with negligible accuracy loss (SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models).
3. GPTQ and AWQ optimize 4-bit weights, in different ways. GPTQ treats quantization as a layer-by-layer reconstruction problem. Using a small calibration set (a sample of typical text; the paper used 128 segments of 2,048 tokens), it quantizes the weights column by column and uses second-order information about the layer (the Hessian, which measures how sensitive the output is to each weight) to adjust the not-yet-quantized weights so they cancel the error. It quantized OPT-175B and BLOOM-176B to 3 or 4 bits in about four GPU hours (GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers). AWQ instead looks at activation magnitudes to find the roughly 1% of weight channels that matter most, scales those channels up before rounding so they lose less precision, and needs no backpropagation or reconstruction at all. Both are weight-only: activations stay in FP16.
What about FP8, MXFP4, NVFP4 and GGUF?
These are the formats you will meet most often today, and they follow the same ideas.
FP8 is an 8-bit floating-point format rather than an integer grid, so each number keeps its own exponent and covers a wide range. E4M3 (4 exponent bits, 3 mantissa bits) reaches ±448 and is the usual choice for inference; E5M2 trades precision for range, up to ±57,344 (FP8 Formats for Deep Learning). The H100 introduced FP8 support in NVIDIA's data center GPUs, and vLLM documents FP8 weights and activations as halving model memory with up to 1.6 times higher throughput and minimal accuracy impact (FP8 W8A8).
Block formats give every small block of weights its own scale, which is what makes 4-bit floats work. MXFP4, from the microscaling (MX) formats, stores blocks of 32 four-bit floats (E2M1) that share one 8-bit power-of-two scale, about 4.25 bits per weight (Microscaling Data Formats for Deep Learning). OpenAI's gpt-oss models ship their mixture-of-experts weights in MXFP4, which is how gpt-oss-120b runs on a single 80 GB GPU (gpt-oss-120b model card). NVFP4, introduced with NVIDIA's Blackwell GPUs, uses smaller blocks of 16 with a finer FP8 scale per block plus one FP32 scale per tensor, and cuts memory about 3.5 times against FP16 (Introducing NVFP4 for Efficient and Accurate Low-Precision Inference).
GGUF is the single-file model format of llama.cpp (and the tools built on it), and its quant names describe the scheme. Q8_0 and Q4_0 are simple blocks of 32 weights with one 16-bit scale each: 8.5 and 4.5 bits per weight. The K in Q4_K_M marks the k-quant family from llama.cpp's k-quants pull request: for Q4_K, super-blocks of 8 blocks of 32 weights with 6-bit scales. The suffix (S, M or L) picks how many sensitive tensors get extra bits. Q4_K_M, the usual default, keeps half of the attention value and feed-forward down-projection tensors at 6 bits and averages about 4.9 bits per weight on Llama 3.1 8B (llama.cpp's quantize README). Part 7 puts these files to work.
Where does the serving stack break when you quantize too aggressively?
Start with speed, because it surprises people. For single-user decoding, weight-only 4-bit is usually the big win. Part 1 showed that decode is limited by reading the weights, and a 4-bit model has a quarter of the bytes to read for every token. GPTQ reported about 3.25 times faster generation on an A100 and 4.5 times on an A6000 with its kernels, and AWQ's TinyChat more than 3 times over Hugging Face's FP16 implementation. Quantizing activations as well (W8A8 or FP8) matters for compute-bound work: long prompts and large batches. There, 8-bit tensor cores double the peak math rate of an H100 compared with FP16.
But quantizing activations requires every activation to be quantized just before the matmul. If the activation scales are fixed in advance, the calibration data used to choose them must cover the real distribution of your traffic. If your serving traffic includes prompts from a different domain, activation ranges can exceed the calibrated thresholds, silently clipping large values. The model does not crash; it just starts producing low-confidence or repetitive text.
The KV cache is a separate choice from the weights. An FP8 KV cache halves its memory and is now a standard option: vLLM's own study found FP8 KV cache and attention cost at most 1 to 2 points of accuracy on short-context tasks and recovered 97 to 98 percent of the baseline on Llama 3.3 70B at 128K tokens, with hybrid models that use small sliding-window layers as the main exception (The State of FP8 KV-Cache and Attention Quantization in vLLM). Caches at 4 bits and below save more but are riskier on long-context work unless the method is built for it, so test them on your own long prompts.
Quick Reference
| Property | Typical value / range |
|---|---|
| FP16 or BF16 weight size | 2 GB per 1B parameters |
| INT8 or FP8 weight size | 1 GB per 1B parameters |
| 4-bit weight size (GPTQ/AWQ, MXFP4) | about 0.5 GB per 1B parameters, plus 3 to 6% for scales |
| Per-channel scheme for weights | One scale per output channel |
| Group-wise 4-bit | One scale (and zero-point) per 128 weights |
| Q4_K_M (GGUF) | about 4.9 bits per weight |
| Activation quantization granularity | Per-tensor or per-token dynamic |
| W8A8 speedup vs FP16 | up to about 1.5× (SmoothQuant 1.56×, vLLM FP8 up to 1.6×) |
| Weight-only 4-bit decode speedup | about 3× or more with good kernels (GPTQ, AWQ papers) |
The numbers below come from the sources linked above: the memory figures from the bit widths themselves, the perplexity rise from AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, and the 99.9% from LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
Frequently Asked Questions
Q: If I quantize a model to INT8 and see 0.2 perplexity increase, does that mean it's safe for my production task?
Not necessarily. Perplexity is an average over all tokens. A small average rise can hide a much larger error rate on the rare tokens that matter to you, such as the ones that make your JSON parser fail. Run your exact prompts through the quantized model and check token-level agreement with the FP16 baseline, especially for rare tokens or structured outputs.
Q: Why can't I just quantize everything with per-channel to solve the outlier problem?
Per-channel works well for weights because they are static and their channels have widely different ranges. For activations you do not know the range ahead of time; a per-channel dynamic scheme would need to compute scales for every matrix multiply at runtime, which is expensive. Instead, most systems use per-tensor or per-token activation scales and attack outliers with methods like SmoothQuant or mixed precision.
Q: Does quantization affect the attention mechanism differently from MLP layers?
Sometimes. Some schemes give a few sensitive tensors extra bits: llama.cpp's Q4_K_M keeps half of the attention value projections at 6 bits, along with half of the feed-forward down-projections, which are MLP tensors. GPTQ and AWQ quantize every linear layer at the same width. The KV cache is a separate setting from the weights, covered above.
Q: How do I know if an accuracy drop is from weight quantization or activation quantization?
Deploy a weight-only variant first. If accuracy is acceptable, then the loss you see in the full W8A8 model comes from activation quantization. That tells you to invest in better calibration data, outlier smoothing (like SmoothQuant), or mixed precision for the activation path without touching the weight quantizer.
Q: Can quantization be combined with speculative decoding or prefix caching?
Yes. Quantizing the large model reduces the memory traffic needed for verification, so speculative decoding runs even faster. The small draft model is usually already small enough that quantization is not the bottleneck, but you can quantize it too if you want. Prefix caching only caches KV entries; if those are quantized, the same tradeoffs apply.
Test yourself
You maintain a production service that serves a quantized Llama 3.1 8B model with W8A8 via the vLLM engine. The FP16 model answers a particular query with "The capital of France is Paris." The quantized model sometimes says "The capital of France is Paris." and sometimes says "The capital of France is Berlin." You suspect activation outliers are to blame.
What is the most likely technical mechanism, and how would you confirm it?
Answer: First rule out the simpler causes. Rerun the prompt at temperature 0 with a batch size of 1. If "Berlin" disappears, the cause is sampling randomness on a flatter distribution, or activation scales that change with whatever else is in the batch, which dynamic per-tensor scales do. If "Berlin" still appears under greedy decoding, compare the two models layer by layer: capture the hidden states for that prompt in FP16 and in W8A8 and diff them, looking for layers where a few channels carry very large values. Such an outlier either stretches the activation scale so the other channels lose precision or, with a fixed calibrated range, gets clipped. Either way the margin between "Paris" and "Berlin" shrinks. If you find those channels, keep them in higher precision (LLM.int8() style), apply SmoothQuant to move the range into the weights, or recalibrate on prompts like this one. Then rerun your prompt set.
Next: Speculative Decoding: A Small Model Drafts, a Big Model Approves. It attacks the same memory bottleneck from another side, getting several tokens out of each pass over the weights.
If you want this kind of breakdown every week, how the machinery inside LLM inference really works, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- LiteRT 8-bit quantization specification (Google)
- Working with Quantized Types (NVIDIA TensorRT documentation)
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al. 2022)
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (Xiao et al. 2022)
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al. 2022)
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al. 2023)
- FP8 Formats for Deep Learning (Micikevicius et al. 2022)
- Microscaling Data Formats for Deep Learning (Rouhani et al. 2023)
- Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (NVIDIA, June 2025)
- gpt-oss-120b model card (OpenAI on Hugging Face)
- llama.cpp quantize README
- llama.cpp k-quants pull request
- FP8 W8A8 (vLLM documentation)
- The State of FP8 KV-Cache and Attention Quantization in vLLM (vLLM blog, April 2026)
