LoRA and QLoRA in depth: rank, alpha, target layers and 4-bit base models

LoRA is the technique that made fine-tuning a large model something an individual can do. You freeze the pretrained weights and train two small matrices beside each large one. This lesson works through what that means numerically, how to choose its settings, and what QLoRA adds. You met the idea briefly in Fine-tuning, LoRA and instruction tuning and the decision of whether to fine-tune at all in When to fine-tune. Everything here runs in plain Python.

#The idea

A weight matrix W of shape out × in takes an input x to Wx. Full fine-tuning changes every entry of W. LoRA keeps W frozen and learns a small correction instead:

h = W·x + (alpha / r) · B·(A·x)

W : out × in    frozen pretrained weights
A : r × in      trainable, starts random (small Gaussian)
B : out × r     trainable, starts at zero
r : the rank, usually 8 to 64
alpha : a scaling constant; the update is multiplied by alpha / r

A squeezes the input down to r numbers and B expands it back. Their product B·A is a full-sized out × in matrix, but it can only have rank r, so it is described by r × (in + out) numbers instead of in × out.

#How small is small?

For a Llama-3.1-8B-shaped model (hidden size 4096, 32 layers, 8 KV heads, feed-forward width 14336), the cost of an adapter is r × (in + out) summed over every matrix you adapt:

hidden, layers, kv_dim, inter = 4096, 32, 1024, 14336
mats = {"q_proj": (hidden, hidden), "k_proj": (hidden, kv_dim), "v_proj": (hidden, kv_dim), "o_proj": (hidden, hidden),
        "gate_proj": (hidden, inter), "up_proj": (hidden, inter), "down_proj": (inter, hidden)}
def lora_params(r, targets): return layers * sum(r * (mats[t][0] + mats[t][1]) for t in targets)

base ≈ 8.03B params
 rank          attention only       all linear layers
    4       3.4M ( 0.042%)       10.5M ( 0.131%)   adapter file ≈     21 MB in BF16
    8       6.8M ( 0.085%)       21.0M ( 0.261%)   adapter file ≈     42 MB in BF16
   16      13.6M ( 0.170%)       41.9M ( 0.522%)   adapter file ≈     84 MB in BF16
   64      54.5M ( 0.679%)      167.8M ( 2.089%)   adapter file ≈    336 MB in BF16
  128     109.1M ( 1.358%)      335.5M ( 4.179%)   adapter file ≈    671 MB in BF16

A rank-16 adapter on every linear layer is about 42 million parameters and an 84 MB file, against 16 GB for the base model in BF16. Three things follow:

• Small to train. Gradients and optimiser state exist only for the adapter, which is where most of the memory saving comes from (how much GPU you need).
• Small to store and ship. You can keep one base model and hundreds of adapters (serving many adapters).
• Cost grows linearly with rank. Doubling r doubles the adapter.

Try it with your own shape:

#The adapter and the merged weights are the same thing

The adapter path computes Wx and B(Ax) separately and adds them. Merging precomputes W + (alpha/r)·B·A once. They give identical outputs:

adapter_out = [w + scale * u for w, u in zip(matvec(W, x), matvec(B, matvec(A, x)))]
merged = [[W[i][j] + scale * BA[i][j] for j in range(d)] for i in range(d)]

adapter path vs merged weights, max difference: 1.11e-16
at initialisation (B = 0) the update is exactly: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

The difference is floating-point rounding. So you can keep adapters separate (small files, hot-swappable, a little extra compute) or merge them into a normal model file with no inference overhead. And because B starts at zero, the first training step begins from exactly the base model's behaviour.

#Why a low-rank update can be enough

The bet LoRA makes is that the change needed to adapt a model to a task is low-rank: it lives in a small number of directions. You can see the effect by building an update with a known hidden rank of 4, adding noise, and approximating it with rank 1, 2, 3 and so on:

rank   energy captured   relative error
   1            42.8%            0.757
   2            67.5%            0.570
   3            86.1%            0.373
   4            98.2%            0.135
   5            98.3%            0.128
   6            98.5%            0.122
   7            98.7%            0.116
   8            98.8%            0.110

Captured energy jumps until the rank reaches the true structure (4), then barely moves: extra rank is fitting noise. Real fine-tuning updates are not this clean, but the same pattern explains why ranks of 8 to 64 work for most tasks, and why very high ranks rarely help. Large domain shifts (for example teaching a model a new programming language from scratch) need more capacity, and that is where full fine-tuning can pull ahead (choosing a method).

#Choosing the settings

Variants you will see in libraries: rank-stabilised LoRA, weight-decomposed LoRA (DoRA), and LoRA+ (different learning rates for the two matrices). Treat them as options to try after a plain LoRA baseline works, and measure the difference on your own evaluation set.

#QLoRA: a 4-bit frozen base

The frozen base is the biggest thing in memory, and it is never updated. QLoRA stores it at about 4 bits per weight while the adapters stay in 16 bits:

• 4-bit NormalFloat (NF4). Weights in a trained network are roughly bell-shaped, with most values near zero. A good 4-bit format puts more of its 16 levels near zero, where the weights are dense.
• Double quantisation. The per-block scale factors are themselves quantised, saving a fraction of a bit more per weight.
• Paged optimisers. Optimiser state can spill to CPU memory during memory spikes instead of crashing.
• Dequantise on the fly. Each layer's 4-bit weights are expanded to BF16 just for the matrix multiply, so gradients flow through to the adapters in higher precision.

The reason a bell-shaped grid helps can be checked directly. Quantise 20,000 normal-looking weights with 16 evenly spaced levels, and with 16 levels placed at equal-probability quantiles of the normal distribution:

4-bit uniform levels   mean squared error 0.0218
4-bit quantile levels  mean squared error 0.0120  (1.8x lower)

NF4 is built on this idea; its exact level values are fixed constants in the library, so this is an illustration of the principle rather than a copy of NF4. In practice QLoRA gets close to the quality of 16-bit LoRA on most tasks, at the cost of slower steps (dequantising every layer) and a small extra quantisation error. If the model already fits in 16 bits on your GPU, plain LoRA is simpler and faster.

#Merge, keep separate, or quantise afterwards

• Merge into a 16-bit base for the best quality and no overhead, then quantise the merged model for serving (quantisation in depth).
• Do not merge into a 4-bit base. Rounding the merged weights back to 4 bits discards part of what the adapter learned. Merge into 16-bit weights first, then quantise.
• Keep adapters separate when you serve many variants from one base.

#Common mistakes

1. Raising rank to fix a data problem. If outputs are wrong because the examples are inconsistent, no rank will fix it (data preparation).
2. Changing alpha and the learning rate without noticing both scale the update.
3. Adapting only attention layers and concluding LoRA cannot learn the task.
4. Using the full-fine-tuning learning rate (around 1e-5) for LoRA, which then barely moves.
5. Merging into a quantised base.
6. Never testing a plain LoRA baseline before trying variants.

#Exercise

For the 8B shape above:

1. How many trainable parameters does a rank-32 adapter on q_proj and v_proj only add?
2. A rank-16 adapter on all linear layers is stored in FP32 instead of BF16. How large is the file?
3. You double the rank from 16 to 32 and keep alpha = 32 fixed. What happens to the scale alpha / r?

Answers:

1. Per layer: q_proj is 32 × (4096 + 4096) = 262,144; v_proj is 32 × (4096 + 1024) = 163,840. Total per layer 425,984, times 32 layers = 13,631,488 (about 13.6M).
2. 41.9M × 4 bytes ≈ 168 MB.
3. The scale halves from 2 to 1, so the update is half as strong. If you want the same strength at the higher rank, raise alpha to 64.

#Going deeper

• Read your library's LoRA configuration and list which module names it targets for your model.
• Next: how much GPU you need to fine-tune.