LoRA, QLoRA and the Training Recipe
Low-rank adapters and 4-bit base models in depth, how much GPU memory a fine-tune needs, loss masking, learning rates, overfitting, forgetting, and how to choose a method.
0 of 4 complete
- ▶ Up next16 min+50 XPLoRA and QLoRA in depth: rank, alpha, target layers and 4-bit base modelsHow low-rank adapters work and why they are so small, with the maths run in code: parameter counts for an 8B model at…
- Pro14 min+50 XPHow much GPU do you need to fine-tune? Memory for full fine-tuning, LoRA and QLoRAWhere training memory goes (weights, gradients, optimiser state, activations), with a calculator and worked tables…
- Pro15 min+50 XPThe training recipe: loss masking, packing, learning rate, epochs and overfittingThe settings that decide whether a fine-tune works, each shown with numbers from code you can run: why only the…
- Pro14 min+50 XPForgetting and choosing a method: full fine-tuning, LoRA, replay and weight interpolationWhy a fine-tuned model loses abilities it had before, shown with a small experiment you can run (a model that learns…