If you have run a model locally, you have used GGUF. It is the format behind llama.cpp and everything built on it, and it is unusually well suited to the job — a single file that contains not just weights but everything else needed to run them. This page covers what is inside one and why the design choices matter.
What it is
A single-file container holding weights, architecture parameters, the tokeniser and the chat template together. No separate config files, no tokeniser directory, no template to configure. Download one file, run it.
The problem it solves
Running a model from the PyTorch ecosystem means assembling several things: weight files, a config.json describing the architecture, tokeniser files, a generation config, and knowledge of which chat template the model expects. Get any of them wrong or missing and the model either fails to load or produces subtly wrong output.
That is reasonable for research, where you are already deep in a framework. It is poor for distribution — and worse, the chat template is the one most often lost. A model loaded with the wrong template still runs and produces noticeably degraded output, with nothing indicating why.
GGUF puts all of it in one file.
| PyTorch distribution | GGUF | |
|---|---|---|
| Files needed | 5–10 | 1 |
| Tokeniser | Separate files | Embedded |
| Chat template | Separate, often lost | Embedded |
| Quantised weights | Framework-specific | Native |
| Loading | Deserialise into RAM | Memory-mapped |
| Runtime dependency | Python and PyTorch | A C++ binary |
The file structure
The design is deliberately simple: fixed header, then a self-describing metadata section, then tensor descriptors, then the raw data. Everything before the tensor data is small, so a loader can read all the structural information without touching the gigabytes that follow.
Why the metadata section matters
GGML — the predecessor — had a fixed header with fixed fields. Supporting a new architecture that needed an extra parameter meant changing the format, which broke every existing file and every existing loader.
GGUF's metadata is an arbitrary key-value map with typed values, namespaced by architecture. A new model family adds newarch.rope_scaling without touching the format specification, and older loaders can still parse the file structure even if they cannot run that architecture.
✅ The embedded chat template is the underrated part
Every instruction-tuned model expects its prompt wrapped in a specific structure — particular delimiter tokens, particular role markers. Use the wrong one and the model still generates, just worse: more repetition, weaker instruction-following, occasional roleplay confusion.
Because the template is stored as a Jinja string in the file's metadata, the runtime reads it from the model rather than requiring you to know it. This single detail eliminates one of the most common and least obvious causes of "this local model seems worse than it should be".
Memory mapping
The tensor data is laid out so it can be memory-mapped rather than read.
Conventional loading reads the file into allocated memory — for a 40GB model, that is 40GB of reading and 40GB of RAM before the first token. Memory mapping instead tells the operating system to map the file into the process's address space. Nothing is read until a page is actually touched, at which point the OS faults it in from disk.
| Read into memory | Memory-mapped | |
|---|---|---|
| Startup | Seconds to minutes | Near-instant |
| RAM needed to start | Full model size | Almost none |
| Two processes, same model | Two copies in RAM | Shared pages |
| Model larger than RAM | Fails | Runs, slowly |
| Under memory pressure | Swaps | OS evicts clean pages |
Two consequences are worth knowing. Running the same model in several processes costs RAM once, because the pages are shared and read-only. And a model larger than physical RAM will load and run — every token touches most of the weights, so it thrashes badly, but "slow" beats "impossible" when you are testing.
This is also why alignment appears in the format. Tensor data is padded to a boundary (32 bytes by default) so mapped regions align with page boundaries and no tensor straddles a page unnecessarily.
⚠️ Memory mapping does not help on GPU
Once layers are offloaded to a GPU, their weights must be copied into VRAM — there is no mapping a file into video memory. The mmap advantage applies to the layers kept in system RAM.
So a fully GPU-offloaded model still needs its weights to fit in VRAM, and startup includes a real transfer. The fast-load property people associate with GGUF is a CPU-inference property.
Decoding the quantisation names
GGUF files are usually distributed in several quantisations with names that look cryptic and are entirely systematic.
| Name | Bits/weight | Size vs FP16 | Quality |
|---|---|---|---|
F16 | 16 | 100% | Reference |
Q8_0 | 8.5 | ~53% | Essentially indistinguishable |
Q6_K | 6.6 | ~41% | Very close to reference |
Q5_K_M | 5.7 | ~36% | Very good |
Q4_K_M | 4.8 | ~30% | The usual recommendation |
Q4_K_S | 4.5 | ~28% | Slightly weaker |
Q3_K_M | 3.9 | ~24% | Noticeable degradation |
Q2_K | 3.4 | ~21% | Substantially degraded |
The bits-per-weight figures exceed the nominal number because quantisation stores scale factors alongside the quantised values — a block of weights shares a scale, and that scale costs bits. A "4-bit" quantisation genuinely averages nearer 4.8 bits once the block metadata is counted.
K-quants improve on the older Q4_0 style by allocating precision unevenly: layers and tensors that matter more to output quality keep more bits. The _S, _M and _L variants choose how generous that allocation is.
💡 The practical rule
Q4_K_M is the default recommendation because it sits at the knee of the curve — roughly 30% of the original size for a quality loss most users cannot detect in normal use. Below Q4 the degradation becomes noticeable; above Q5 you pay significant size for diminishing gains.
The more consequential choice is usually which model rather than which quantisation: a larger model at Q4 generally beats a smaller model at Q8 of the same total file size.
Inspecting a GGUF file
The metadata dump is genuinely useful before downloading a large file — it tells you the architecture, the trained context length, the quantisation and whether a chat template is present. A GGUF missing its chat template is worth avoiding.
Working out what will fit
| Model | Q4_K_M size | Comfortable on |
|---|---|---|
| 3B | ~2.0 GB | 8 GB RAM, most laptops |
| 7–8B | ~4.5 GB | 16 GB RAM or 8 GB VRAM |
| 13B | ~7.9 GB | 16 GB RAM or 12 GB VRAM |
| 34B | ~20 GB | 32 GB RAM or 24 GB VRAM |
| 70B | ~42 GB | 64 GB RAM or 2× 24 GB VRAM |
The KV cache is the part people forget, and it scales with context length. Running a 7B model at 32k context rather than 4k can add several gigabytes — enough to turn a model that fits into one that does not, with the failure appearing only once a conversation gets long.
GGUF and safetensors
| GGUF | Safetensors | |
|---|---|---|
| Purpose | Inference | Storing tensors |
| Contains tokeniser | Yes | No |
| Contains chat template | Yes | No |
| Quantised layouts | Native | Not really |
| Ecosystem | llama.cpp and descendants | PyTorch, Hugging Face |
| Training | No | Yes |
| Safe to load | Yes | Yes |
They are not competitors. Safetensors is a container for tensors and nothing else, used where a framework already supplies the surrounding configuration. GGUF is a complete inference package, used where you want one file that just runs.
Both share the most important property: neither executes code on load. That distinguishes them from the pickle-based .pt and .bin files that preceded them, which do — and which remain a genuine security problem worth understanding separately.
Verifying a large model download?
Generate SHA-256 checksums in your browser and compare against the publisher's — no upload, works on files of any size.
Open the Hash Generator →Summary
- GGUF is one file containing weights, tokeniser and chat template.
- Extensible key-value metadata is why it replaced GGML — new architectures no longer break the format.
- The embedded chat template quietly prevents a common cause of degraded local output.
- Memory mapping gives near-instant loading and lets several processes share one copy.
- mmap does not apply to GPU-offloaded layers — those must fit in VRAM.
- Q4_K_M is the sensible default, at roughly 30% of full size.
- Bits-per-weight exceeds the nominal number because block scale factors cost bits.
- Budget for the KV cache — it grows with context and is what makes long conversations fail.
Frequently Asked Questions
What is a GGUF file?
A single-file container holding a model's weights, its architecture parameters, its tokeniser and its chat template together. It was designed for the llama.cpp ecosystem so that one file is everything needed to run a model, with no accompanying config or tokeniser files.
Why did GGUF replace GGML?
GGML had no extensible metadata, so every new architecture or parameter required changing the format itself and breaking existing files. GGUF stores arbitrary key-value metadata, which means new architectures can add what they need while older loaders still parse the structure.
What does Q4_K_M mean?
Roughly 4 bits per weight, using the K-quant method, in the medium variant. Q is quantised, the number is the nominal bit width, _K indicates K-quants which allocate precision unevenly across a block, and _S, _M or _L select how much of the model keeps higher precision.
How can a model larger than my RAM still load?
Memory mapping. The file is mapped into the address space rather than read into memory, so the operating system pages in only the parts actually touched and evicts them under pressure. It works, and performance degrades sharply once the working set exceeds physical RAM.
Is GGUF better than safetensors?
They solve different problems. Safetensors is a safe, fast tensor container used for training and for the PyTorch ecosystem, and it stores nothing else. GGUF is a self-contained inference format that also carries the tokeniser, chat template and quantised weight layouts. Use safetensors for training, GGUF for local inference.