Fine-tuning a 7-billion-parameter model the obvious way produces another 7-billion-parameter model — 14GB, for a change that might amount to "answer in our house style". LoRA sidesteps that with a piece of linear algebra that is genuinely elegant, and the resulting files are small enough to email. It also has limits that are widely ignored until someone tries to teach a model facts with one.
The trick in one line
Instead of learning a full weight update ΔW (huge), learn two thin matrices A and B whose product approximates it. Store only A and B.
A 4096×4096 update is 16.8M numbers. As rank-8 matrices it is 65K — 256× fewer.
The arithmetic
Fine-tuning adjusts a weight matrix: W' = W + ΔW. Both matrices are the same size, so storing ΔW costs as much as storing the model.
LoRA's observation is that ΔW, for most adaptations, is low rank — it does not need the full expressive capacity of a dense matrix, because the adaptation is a relatively simple transformation. So factor it:
Applied across the layers that get adapted — usually the attention projections rather than every matrix in the model — a 7B fine-tune becomes a file of roughly 10–60MB depending on rank and coverage.
| Approach | 7B model | Storage per variant |
|---|---|---|
| Full fine-tune (fp16) | 14 GB | 14 GB each |
| LoRA, rank 8 | base + adapter | ~15 MB |
| LoRA, rank 32 | base + adapter | ~60 MB |
| LoRA, rank 128 | base + adapter | ~240 MB |
The operational consequence is larger than the storage one. Fifty full fine-tunes is 700GB and fifty separate deployments. Fifty LoRAs is one base model in memory plus 750MB of adapters, swappable per request.
Why training is also cheaper
The file size gets the attention, but the training saving is what made LoRA ubiquitous. Only A and B receive gradients — the base weights are frozen.
That is the difference between needing a cluster and needing one card, which is why fine-tuning stopped being something only well-resourced labs did.
💡 Initialisation is what makes it stable
B is initialised to zeros and A to small random values. So at step zero, B × A = 0 and the adapter has no effect whatsoever — the model behaves exactly like the base.
Training then moves away from zero gradually. This means a LoRA run always starts from a known-good model rather than a perturbed one, which is a large part of why LoRA training is so much more stable than full fine-tuning at small data sizes.
Rank and alpha
The two numbers you choose, and the two most often misunderstood.
Rank (r) is capacity. Higher rank means the adapter can express more complex changes, at proportionally more parameters.
| Rank | Suits |
|---|---|
| 4–8 | Style, tone, output format |
| 16–32 | Most task fine-tuning |
| 64–128 | Substantial behaviour change, larger datasets |
| 256+ | Rarely justified — consider full fine-tuning |
Alpha (α) is a scaling factor. The adapter is applied as:
The division by r is the point: it means changing rank does not change the effective magnitude of the update, so you can raise rank without having to retune the learning rate. A common convention is α = 2r, and if you take one thing from this section it should be that α and r are not independent knobs — what matters at inference is their ratio.
Merged or separate
Two deployment modes, and the choice has real consequences.
Merge when you ship one specialised model and want it to behave exactly like any other model. Keep separate when you serve many variants — the memory saving is enormous, because one base model in VRAM serves every adapter, and swapping is a matter of megabytes.
⚠️ Merging into a quantised base loses precision
If your base model is 4-bit quantised and your adapter is fp16, merging forces the sum back through quantisation. The adapter's fine adjustments are exactly the kind of small values that quantisation rounds away, and you can lose much of what you trained.
This is why QLoRA setups — quantised base, fp16 adapter — typically keep the adapter separate at inference rather than merging. If you must merge, merge into the full-precision base and quantise afterwards.
What is actually in the files
A LoRA adapter is normally distributed as two files:
target_modules is worth reading before using someone else's adapter. An adapter trained on q_proj and v_proj only touches those; one trained on all four attention projections plus the MLP is a much more thorough adaptation and a much larger file. The two are not interchangeable, and the config is the only place that distinction is recorded.
The weights themselves are stored as safetensors, so an adapter carries the same no-code-execution property as a modern base model.
What LoRA cannot do
The honest section, and the one that saves the most wasted effort.
| Goal | LoRA suitable? |
|---|---|
| Output format and structure | Excellent |
| Tone, voice, house style | Excellent |
| Task following, instruction shape | Good |
| Domain vocabulary and phrasing | Good |
| Teaching new facts | Poor — use retrieval |
| Adding a new language | Poor |
| Fundamentally new capability | No |
The pattern: LoRA is very good at changing how a model responds and poor at changing what it knows. The low-rank constraint is a limit on how much new information the update can carry, and facts are information-dense in exactly the way the decomposition compresses away.
The common failed project is fine-tuning an adapter on a company knowledge base to make the model "know" it. What you generally get is a model that has learned the style of your documents and still invents the details. Retrieval is the right tool for facts — the model looks them up rather than trying to store them in a rank-16 matrix.
🚨 Adapters are welded to their base model
A LoRA is a set of deltas against specific matrices of specific shapes in a specific model. It is not portable to a different base, a different size, or in some setups even a differently quantised copy of the same model.
Record the exact base model and revision alongside every adapter you train. When the base is updated — and bases get updated — your adapters do not carry over, and an adapter whose base you can no longer identify is scrap.
Inspecting adapter configs?
Format and validate JSON entirely in your browser — nothing is uploaded to a server.
Open JSON Formatter →Summary
- ΔW = B × A with a small inner rank replaces a full weight update.
- Rank 8 on a 4096×4096 matrix is 256× fewer parameters.
- Training memory falls further than file size — frozen base means no gradients or optimiser state for it.
- B starts at zero, so training begins from the exact base behaviour.
- α and r act as a ratio. Changing rank alone does not change update strength.
- Merge for a single shipped model; keep separate to hot-swap many.
- Do not merge fp16 adapters into quantised bases — quantisation eats the adjustment.
- LoRA changes behaviour, not knowledge. For facts, use retrieval.
Frequently Asked Questions
What is a LoRA file?
A small file containing only the low-rank matrices learned during LoRA fine-tuning, rather than a full copy of the model. Instead of storing updated weights for every parameter, it stores two thin matrices per adapted layer whose product approximates the weight change, which is why it is typically tens of megabytes against tens of gigabytes.
How does LoRA make files so much smaller?
By representing a weight update as the product of two thin matrices instead of one full one. A 4096 by 4096 update holds about 16.8 million numbers; the same update as two rank-8 matrices holds about 65,000 — roughly 256 times fewer, and that ratio is the whole saving.
What do rank and alpha mean in LoRA?
Rank is the inner dimension of the two matrices and sets how much the adapter can express — higher rank means more capacity and a larger file. Alpha is a scaling factor applied as alpha divided by rank, which controls how strongly the adapter influences the base model at inference time.
Can I use a LoRA with any base model?
No. An adapter is trained against specific weight matrices with specific shapes, so it only applies to the exact base model it was trained on. Using it with a different model, or even a different quantisation in some setups, produces either an error or degraded nonsense.
What can't LoRA do?
It adapts behaviour far better than it adds knowledge. Style, format, tone and task-following respond well to LoRA, while teaching genuinely new facts is limited by the low-rank constraint and usually better handled with retrieval. If a model does not know something, an adapter is rarely the right fix.