Local LLM advice tends to arrive as hardware recommendations, which is the least useful form it could take — the specifications change, and the reasoning behind them does not. One equation predicts almost everything about local inference performance, and it is not about the processor.
The equation that matters
Generating one token requires reading every weight from memory. So throughput is set by how fast memory can be read, not by compute. A faster processor on the same memory changes very little.
Why bandwidth, not FLOPs
Single-stream generation is memory-bound, and understanding why makes the rest of local inference predictable.
This explains a set of otherwise confusing observations:
- A fast CPU with ordinary DDR5 disappoints. Dual-channel DDR5 gives roughly 60–90 GB/s; the processor is idle waiting for weights.
- GPUs are fast because of memory, not cores. Modern GPU VRAM runs at hundreds of GB/s to over a terabyte — often ten times system RAM.
- Quantisation speeds up generation. Halving the model halves the bytes read per token, so it roughly doubles throughput. The speed gain is a side effect of the size reduction.
- Prompt processing behaves differently. Reading a long prompt is compute-bound and parallel, which is why a long prompt processes quickly and then generation crawls.
| Memory type | Rough bandwidth | 7B at 4-bit (~4GB) |
|---|---|---|
| DDR4 dual channel | ~40 GB/s | ~10 tok/s |
| DDR5 dual channel | ~80 GB/s | ~20 tok/s |
| Apple Silicon (Pro/Max) | ~200–500 GB/s | 50–100+ tok/s |
| Consumer GPU VRAM | ~400–1000 GB/s | 100–200+ tok/s |
These are ceilings, not measurements — real throughput lands below them because of cache behaviour, sampling overhead and the KV cache. But the ratios hold, and they are what you should reason with.
⚠️ Split loading is a cliff, not a slope
If a model does not fit in VRAM, most runtimes will place some layers on the GPU and the rest in system RAM. This works, and performance collapses — every token now waits on the slowest memory in the chain.
A model that fits entirely in VRAM can be five to ten times faster than the same model with a few layers spilled. When choosing quantisation, fitting fully in fast memory beats a higher-quality quant that does not, almost every time.
How much memory
The full arithmetic is in parameters to gigabytes to VRAM. The practical version for local use:
| Model | 4-bit size | Comfortable with |
|---|---|---|
| 3B | ~2 GB | 8 GB RAM, any recent machine |
| 7–8B | ~4–5 GB | 16 GB RAM or 8 GB VRAM |
| 13–14B | ~8 GB | 16 GB VRAM or 24 GB RAM |
| 32B | ~19 GB | 24 GB VRAM or 32 GB unified |
| 70B | ~40 GB | 48 GB VRAM or 64 GB unified |
Add headroom for context. The KV cache grows with conversation length, and a long session can want several more gigabytes than the model itself — the figures above assume moderate context, not a filled 128k window.
The Apple Silicon case
Unified memory is genuinely advantageous here rather than a marketing point. CPU and GPU share one pool at high bandwidth, so a 64GB Mac can load a 70B model that would otherwise need a data-centre card. You pay for it in peak throughput — a high-end NVIDIA GPU is still faster on models that fit its VRAM — but capacity per pound is strongly in Apple's favour, and capacity is the binding constraint for most people.
Choosing a quantisation
Quantisation is where most local setups are won or lost, and the decision is simpler than the option list suggests.
| Level | Size vs fp16 | Verdict |
|---|---|---|
| 8-bit | 50% | Effectively lossless; use if it fits |
| 6-bit | ~38% | Very close to lossless |
| ~4-bit (K-quant, medium) | ~28% | The default sweet spot |
| 3-bit | ~21% | Noticeable degradation |
| 2-bit | ~16% | Usually not worth running |
Two rules cover nearly every case. A larger model at 4-bit beats a smaller model at 8-bit when both fit — parameter count buys more capability than precision does. And the best quant is the largest one that fits entirely in fast memory, because of the split-loading cliff above.
Mixed schemes — where different tensors get different precision, keeping sensitive layers higher — are why modern 4-bit quantisation holds up as well as it does. The mechanism is covered in what quantisation actually does.
Disk, and the part nobody plans for
Models are also downloaded once and read constantly. On an SSD, memory-mapped loading means a model can start responding almost immediately; on a mechanical drive, first load is slow enough to be unpleasant. Keep models on your fastest drive.
The runtimes
| Tool | Suits |
|---|---|
| Ollama | Getting running in one command |
| llama.cpp | Control, embedding in your own code |
| LM Studio | A GUI, browsing and trying models |
| vLLM | Serving many concurrent users on GPUs |
| MLX | Apple Silicon specifically |
Most of these sit on GGUF and llama.cpp underneath, so the model files are largely interchangeable between them. vLLM is the exception — it targets high-throughput GPU serving with batching, and is the right answer for a shared endpoint rather than a personal one.
🚨 Specifics here will age; the reasoning will not
Model sizes, quantisation naming, runtime capabilities and hardware figures all move quickly, and this article is written from a point in time. Treat the tables as worked examples rather than current specifications.
The durable parts are the mechanisms: throughput follows memory bandwidth divided by model size, fitting entirely in fast memory beats a better quant that spills, and parameter count buys more than precision. Those hold regardless of what ships next year — check current figures for anything you are about to buy.
When local is actually the right call
An honest accounting, because the economics are frequently misrepresented in both directions.
| Reason | Does local win? |
|---|---|
| Data must not leave the machine | Yes — the strongest case |
| Offline or air-gapped operation | Yes |
| High sustained volume | Yes, past a real threshold |
| Predictable fixed cost | Yes |
| Experimenting, learning the stack | Yes |
| Saving money at moderate use | Usually not |
| Matching frontier model quality | No |
The last two rows are where expectations break. A GPU capable of running a large model comfortably costs more than several years of moderate API use, so "saving money" rarely survives the arithmetic unless volume is genuinely high. And a model you can run on one machine is not equivalent to the largest hosted models — capable, often sufficient, and not the same thing.
Privacy is the case that stands up without qualification. If the data cannot leave the machine, no API pricing changes that, and local inference is the only option — the same reasoning that makes browser-based file tools worth using for sensitive documents.
Handling files you would rather not upload?
Convert, compress and clean files entirely in your browser — nothing leaves your device.
Browse all tools →Summary
- tokens/sec ≈ memory bandwidth ÷ model size. Bandwidth, not compute.
- Prompt processing is compute-bound; generation is memory-bound.
- Spilling out of fast memory is a cliff — five to ten times slower, not slightly.
- 7B at 4-bit runs comfortably on 16GB RAM or 8GB VRAM.
- Apple unified memory buys capacity; NVIDIA buys peak speed.
- 4-bit K-quant is the default; a bigger model at 4-bit beats a smaller one at 8-bit.
- Budget far more disk than you expect.
- Local wins on privacy and offline use, rarely on price at moderate volume.
Frequently Asked Questions
What hardware do I need to run an LLM locally?
Enough fast memory to hold the model, which matters more than raw compute. A 7B model at 4-bit quantisation needs about 5GB and runs on most modern laptops; a 70B model at 4-bit needs around 40GB and wants a high-VRAM GPU or a large unified-memory Mac.
Why is local LLM generation so slow on CPU?
Because generating each token requires reading every weight in the model from memory, so speed is set by memory bandwidth rather than processing power. A rough estimate of tokens per second is memory bandwidth in GB/s divided by model size in GB, which is why a fast CPU with slow RAM disappoints.
Which quantisation level should I use?
Around 4-bit with a K-quant mixed scheme is the usual sweet spot — roughly a quarter the size of fp16 with quality loss most people cannot detect in normal use. Go higher only if you have memory to spare, and lower only if the alternative is not running the model at all.
Is a Mac good for running LLMs locally?
Apple Silicon is unusually well suited because CPU and GPU share one pool of high-bandwidth unified memory, so a machine with 64GB can load models that would need an expensive dedicated GPU on a PC. Peak throughput still trails high-end NVIDIA cards, but the capacity per pound is favourable.
Is running an LLM locally cheaper than using an API?
Only at sustained volume, or when privacy is the actual requirement. Hardware capable of running a large model costs more than years of moderate API use, and a local model of a size most people can run is not equivalent to a frontier hosted model. Local wins on privacy, offline use and predictable cost, not usually on raw price.