"Will this model fit on my card" is the most common question in local inference and the one most often answered wrong, because the obvious calculation gives you the file size and the file size is not the requirement. The gap between the two is where people download 40GB and discover it will not load.
The two calculations
File size = parameters × bytes per parameter. A 7B model at fp16 (2 bytes) is ~14GB.
Memory to run it = weights + KV cache + activations + overhead. At short context that is roughly weights × 1.2. At long context the KV cache dominates and can exceed the weights.
Parameters to gigabytes
One multiplication, and the only variable is how many bytes each parameter occupies.
| Precision | Bytes/param | 7B | 13B | 70B |
|---|---|---|---|---|
| fp32 | 4 | 28 GB | 52 GB | 280 GB |
| fp16 / bf16 | 2 | 14 GB | 26 GB | 140 GB |
| 8-bit | 1 | 7 GB | 13 GB | 70 GB |
| ~5-bit | 0.65 | 4.8 GB | 9 GB | 48 GB |
| ~4-bit | ~0.5 | 3.8 GB | 7.4 GB | 39 GB |
Two notes on the lower rows. Quantised sizes are approximate because real quantisation schemes are not uniform — they store some tensors at higher precision and add per-block scaling factors, so a "4-bit" model averages slightly above 4 bits per weight. And fp32 is listed for completeness; almost nothing is distributed that way now, since bf16 gives the same usable range at half the size.
💡 Why bf16 rather than fp16
Both are 2 bytes. They spend those bytes differently: fp16 gives more precision and a narrow exponent range, bf16 gives less precision and the same range as fp32.
For neural networks, range matters more than precision — overflowing to infinity breaks training, small rounding errors do not. That is why bf16 became the default for modern models, and why a bf16 file converted naively to fp16 can produce garbage on models with large activation values.
The KV cache, which is what actually gets you
Weights are static. The KV cache is not, and it is the reason a model that loads happily will run out of memory forty minutes into a conversation.
When generating, the model attends to every previous token. Recomputing their key and value vectors each time would be quadratic work, so they are cached. That cache grows by a fixed amount per token, forever.
Worked, for a 7B-class model with 32 layers, 32 heads of 128 dimensions, at fp16:
That progression is the single most useful thing on this page. A 14GB model at 128k context needs roughly 78GB of memory, and 64GB of it is cache. This is why long-context inference is expensive in a way that has nothing to do with model size, and why providers price long contexts the way they do.
⚠️ Grouped-query attention changes this substantially
Most current models do not use one key-value head per query head. Grouped-query attention shares KV heads across groups of query heads, and the cache shrinks by exactly that ratio.
A model with 32 query heads and 8 KV heads has a quarter the cache of the figures above. Check num_key_value_heads in the model's config rather than assuming it equals the head count — getting this wrong overestimates memory by 4× or more, and it is the most common error in these calculations.
Everything else
Three smaller terms, none negligible:
- Activations — working memory for the current forward pass. Roughly proportional to batch size and hidden dimension; typically a few hundred MB to a couple of GB for single-stream generation.
- Framework and driver overhead — the CUDA context alone is several hundred MB before your model exists. Budget 1–2GB.
- Fragmentation — allocators do not pack perfectly. Assume you cannot use the last 5–10% of nominal VRAM.
That last line is the practical lesson. "14GB model, 16GB card" looks comfortable and is not. The usable rule: a card fits a model roughly 60–70% of its VRAM at fp16 with modest context.
What fits where
| VRAM | Comfortable at fp16 | Comfortable at 4-bit |
|---|---|---|
| 8 GB | 3B | 7B |
| 12 GB | 7B (short context) | 13B |
| 16 GB | 7B | 13B, long context |
| 24 GB | 13B | 32B |
| 48 GB | 32B | 70B |
| 80 GB | 70B | 70B, very long context |
The pattern worth internalising: 4-bit quantisation moves you up roughly one hardware tier. That is the entire practical argument for quantisation, and it is a strong one — the quality cost of 4-bit is modest, and the alternative is not running the model at all.
Quantisation saves less VRAM than file size suggests
A common disappointment. Quantising from fp16 to 4-bit cuts the file to a quarter, so people expect VRAM to quarter too. It does not, because the KV cache is usually kept at higher precision.
KV cache quantisation exists and helps, but it degrades quality more noticeably than weight quantisation — the cache is what the model attends to, and errors there compound across the sequence. Most setups keep it at fp16 or fp8 for that reason.
A note on mixture-of-experts
MoE models break the parameters-to-memory relationship in a way that trips people up. A model described as 8×7B does not have 7B of weights — it has close to 47B, all of which must be in memory, even though only about 13B are active for any given token.
So MoE buys you speed at a given quality, not memory savings. Size the hardware on total parameters and expect throughput closer to the active count.
🚨 These figures move — check the config file
Architectures change. Head counts, GQA ratios, layer counts, vocabulary sizes and quantisation schemes all vary by model and generation, and the numbers here are worked examples rather than constants.
The durable part is the method: parameters × bytes for the file, plus the KV formula for the cache. Read config.json for the real num_hidden_layers, num_key_value_heads and hidden_size, and run the arithmetic on those.
Inspecting model config files?
Format and explore JSON configs entirely in your browser — nothing is uploaded to a server.
Open JSON Formatter →Summary
- File size = parameters × bytes per parameter. 7B at fp16 = 14GB.
- bf16 beat fp16 on range, which matters more than precision for networks.
- The KV cache grows linearly with context and can exceed the weights entirely.
- Check
num_key_value_heads— grouped-query attention cuts the cache several-fold. - Add ~1.5GB overhead and expect to lose 5–10% to fragmentation.
- A card comfortably fits a model at 60–70% of its VRAM at fp16.
- 4-bit quantisation moves you up one hardware tier, but saves less VRAM than file size at long context.
- MoE memory follows total parameters, speed follows active ones.
Frequently Asked Questions
How do I calculate a model's file size from its parameter count?
Multiply the parameter count by the bytes used per parameter. At fp16 or bf16 that is 2 bytes, so a 7-billion-parameter model is about 14GB. At fp32 it is 4 bytes and doubles; at 4-bit quantisation it is roughly half a byte and the same model lands near 4GB.
How much VRAM do I need to run a model?
The weights are the floor, not the requirement. Add the KV cache, which grows with context length and batch size, plus activations and framework overhead. For short contexts, weights times 1.2 is a workable estimate; at long contexts the KV cache can exceed the weights entirely.
What is the KV cache and why does it use so much memory?
It stores the key and value vectors for every token already processed, so the model does not recompute them for each new token. Its size grows linearly with sequence length, so a long conversation or document can require more memory for the cache than for the model itself.
Why does a 7B model need more than 14GB of VRAM?
Because the 14GB is only the weights. The CUDA context and framework take a fixed amount, activations need working space during each forward pass, memory fragmentation wastes some, and the KV cache grows as you generate. In practice a 7B model at fp16 wants around 16-18GB for comfortable use.
Does quantisation reduce VRAM as much as file size?
It reduces the weights proportionally, but the KV cache is often kept at higher precision, so total memory falls by less than the file size suggests. Quantising a model to a quarter of its size might halve total VRAM at long context rather than quartering it.