Quantisation is what makes running a capable model on a laptop possible. It is also frequently described in a way that obscures what is actually happening — "reduces precision" is true and explains nothing about why some quantisations are nearly lossless and others break the model.
The core idea
Store each weight in fewer bits. A 16-bit float becomes a 4-bit integer plus a shared scale factor for its block, so the original value can be approximately reconstructed. Size falls roughly 4×; quality falls far less than that, because the scale factors preserve most of the useful information.
How a block is quantised
Weights are not quantised individually — they are quantised in blocks, typically 32 or 64 at a time. This is the detail that makes the whole thing work.
Each weight is now an approximation. The error is bounded by half a quantisation step, and because the scale is fitted to this block's range, blocks of small weights get fine steps and blocks of large weights get coarse ones. Precision follows the data.
This is why block size matters. Smaller blocks track local variation better and cost proportionally more scale-factor storage.
Why "4-bit" is not 4 bits
So the size figures in a quantisation table are always higher than the nominal bit width. It is not marketing imprecision — the metadata is genuinely part of the file, and it is what keeps quality acceptable.
K-quants: precision where it matters
Older quantisation applied the same bit width to every weight in the model. K-quants recognise that not all weights matter equally.
Attention weights and the layers nearest the input and output have a disproportionate effect on output quality. Feed-forward weights in middle layers tolerate more error. K-quants allocate accordingly — some tensors keep 5 or 6 bits while the bulk drops to 4.
| Variant | Allocation | Typical bits/weight |
|---|---|---|
Q4_K_S | Small — most tensors at 4 bits | 4.5 |
Q4_K_M | Medium — attention layers keep 6 bits | 4.8 |
Q5_K_M | More tensors at higher precision | 5.7 |
Q6_K | Uniformly high | 6.6 |
The gain is real: Q4_K_M is meaningfully better than a uniform 4-bit quantisation of the same size, because the extra bits went where they do the most good.
What degrades first
Quality loss is not uniform across tasks, and knowing the order helps you choose.
| Capability | Sensitivity |
|---|---|
| Casual conversation | Very robust — survives to Q3 |
| Summarisation | Robust |
| Factual recall | Moderate — rare facts degrade first |
| Instruction following | Moderate |
| Code generation | Sensitive |
| Multi-step reasoning | Very sensitive |
| Long-context retrieval | Very sensitive |
| Non-English languages | Sensitive — less represented in training |
⚠️ Reasoning degrades before you notice it
A chain of reasoning compounds error at each step. A small per-step degradation that is invisible in a single response becomes a wrong conclusion by step six.
So a heavily quantised model can chat convincingly and fail at exactly the tasks you deployed it for. If your workload is agentic, code-generating or multi-step, test at that task rather than trusting a conversational impression.
Perplexity, and what it hides
Quantisation quality is usually reported as perplexity — roughly, how surprised the model is by a held-out text. Lower is better, and a Q4_K_M model typically sits a fraction of a percent above the original.
That number is genuinely useful and it measures average next-token prediction on ordinary prose. It does not measure whether the model still produces valid JSON, still follows a ten-step instruction, or still retrieves the right fact from 8,000 tokens of context.
✅ Test on your actual task
Take thirty representative inputs from your workload, run them through the full-precision and quantised models, and compare outputs directly. Half an hour of this tells you more than any published perplexity table.
Pay particular attention to output format adherence. Quantised models degrade at reliably producing structured output before they degrade at sounding coherent, and format failures are what break pipelines.
Bigger model or higher precision?
| Option | Size | Generally |
|---|---|---|
| 7B at Q8 | ~7.2 GB | Baseline |
| 13B at Q4_K_M | ~7.9 GB | Better |
| 7B at Q4_K_M | ~4.5 GB | Worse than either, and much smaller |
| 34B at Q3_K_M | ~16 GB | Usually beats 13B at Q6 |
At roughly equal file size, more parameters at lower precision generally wins. Capability scales with parameter count more strongly than with per-weight precision, and a larger model has more redundancy to absorb quantisation error.
The relationship breaks down at the extremes — a 70B model at 2 bits is not better than a 13B at 5 — but across the normal range it is a reliable guide, and it is the opposite of most people's instinct.
Why it also runs faster
Token generation is memory-bandwidth-bound, not compute-bound. Producing each token requires reading essentially every weight from memory, and that transfer dominates.
Halving the model roughly doubles throughput. The arithmetic to unpack 4-bit values costs something, and it is far cheaper than the memory traffic it avoids.
This also explains why GPU inference is so much faster: high-bandwidth video memory runs at hundreds of gigabytes per second against system RAM's tens.
Quantisation approaches
| Method | Needs data? | Notes |
|---|---|---|
| Round-to-nearest | No | Simple; what basic GGUF quants do |
| K-quants | No | Uneven allocation by tensor importance |
| Importance matrix | Yes | Calibration data identifies which weights matter |
| GPTQ | Yes | Layer-wise, compensates for error as it goes |
| AWQ | Yes | Protects weights with large activations |
| QAT | Full training | Best quality; requires retraining |
Calibration-based methods run sample text through the model to observe which weights actually influence outputs, then protect those. The result is measurably better at the same bit width — an importance-matrix Q4 typically beats a plain Q4 noticeably, and at very low bit widths the difference is large.
The trade-off is that calibration data introduces a bias. Calibrate on English prose and the model may degrade more on code or other languages than the headline numbers suggest.
Choosing
- Start with Q4_K_M. The right default for almost everyone.
- If it fits, try Q5_K_M or Q6_K for precision-sensitive work.
- Prefer a bigger model at Q4 over a smaller one at Q8 at equal size.
- Avoid Q2 and Q3 unless you have measured them on your task.
- Leave headroom for the KV cache — it grows with context and is what makes long conversations fail.
- Test on your workload, especially structured output.
Verifying a quantised model download?
Generate SHA-256 checksums in your browser to confirm a multi-gigabyte file transferred intact.
Open the Hash Generator →Summary
- Weights are quantised in blocks with a shared scale factor, so precision follows the data.
- "4-bit" is really 4.5–4.8 once block metadata is counted.
- K-quants allocate more bits to layers that matter, which is why they beat uniform quantisation.
- Reasoning and code degrade first; casual conversation degrades last.
- Perplexity misses format adherence and long-context retrieval.
- Bigger model at Q4 beats smaller at Q8 at equal size.
- Quantisation speeds inference up because generation is bandwidth-bound.
- Q4_K_M is the default. Test anything more aggressive.
Frequently Asked Questions
What does quantisation do to a model?
It stores each weight using fewer bits — 4 or 8 instead of 16 — which shrinks the file and the memory needed to run it. The values become approximations of the originals, so the model's outputs shift slightly. Done well the shift is small enough that most users cannot detect it.
Why is 4-bit quantisation actually 4.8 bits per weight?
Because weights are quantised in blocks and each block stores a scale factor alongside its values. That metadata is real storage. A block of 32 weights at 4 bits plus a 16-bit scale averages about 4.5 bits, and K-quants add further per-block data that pushes it higher.
Which quantisation level should I use?
Q4_K_M for most local use — around 30% of the original size with quality loss most people cannot detect in normal conversation. Go to Q5 or Q6 if you have the memory and the task is precision-sensitive. Below Q4 the degradation becomes noticeable, particularly on reasoning and code.
Is a bigger model at low precision better than a smaller one at high precision?
Usually yes, at equal file size. A 13B model at Q4 generally outperforms a 7B at Q8 despite occupying similar space, because parameter count contributes more to capability than per-weight precision does. This holds until quantisation gets very aggressive.
Does quantisation make a model faster?
Often, yes — mainly because inference is memory-bandwidth-bound rather than compute-bound. Moving fewer bytes from memory to the processor for every token is a direct speedup, which is why a quantised model can run faster even with the overhead of unpacking values.