Set up your fine-tuning environment: GPUs, libraries and a check script

Fine-tuning is the one part of this site where hardware and library setup is the most common blocker. This lesson gets you to a known-good environment.

#1. Choose where to run

Rule of thumb: do the data work on your laptop, and rent a GPU only for the hours you train.

#2. Will it fit? The memory arithmetic

Memory for a LoRA run has four parts: the frozen base weights, the adapter with its gradients and optimiser state, the activations, and some overhead (the full calculation).

params = 8e9                                  # an 8B model
w16 = params * 2   / 2**30
w4  = params * 0.5 / 2**30
print(f"weights 16-bit: {w16:.1f} GiB, 4-bit: {w4:.1f} GiB")      # 14.9 GiB, 3.7 GiB

ad = 42e6                                     # a rank-16 adapter has about 42M parameters
adapter = ad * (4 + 4 + 4 + 4) / 2**30        # weights, gradients and two Adam states, all in fp32
print(f"adapter + gradients + optimiser: {adapter:.2f} GiB")     # 0.63 GiB

So a QLoRA run (4-bit base) of an 8B model needs about 3.7 GiB for weights plus under 1 GiB for the adapter, and the rest is activations, which depend on batch size and sequence length. The calculator lesson estimates about 5.6 GB at batch size 1 and sequence length 2048, rising with batch size. That is why an 8 GB card can train a QLoRA of an 8B model with a small batch and a short sequence, while 16-bit LoRA wants a 24 GB class card.

If it does not fit: lower the sequence length, the batch size (use gradient accumulation to keep the effective batch), the rank, or move to 4-bit.

#3. Install

Create an isolated environment and install the libraries the labs use:

python3 -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\activate
pip install torch transformers datasets peft trl accelerate
pip install bitsandbytes             # only for 4-bit QLoRA on NVIDIA GPUs

Install the PyTorch build that matches your hardware (the PyTorch site has a selector for CUDA versions). Then record the versions you got:

pip freeze | grep -iE "torch|transformers|peft|trl|datasets|accelerate|bitsandbytes"

These libraries change quickly, and a lab that worked last month can break on a newer release. Keep the versions that worked in a requirements.txt, and when something fails after an upgrade, check the library's release notes first.

#4. A check script

Save as envcheck.py:

import sys, importlib.util, shutil

print("Python", sys.version.split()[0])
for name in ("torch", "transformers", "peft", "trl", "datasets", "bitsandbytes", "accelerate"):
    spec = importlib.util.find_spec(name)
    ver = "not installed"
    if spec:
        try:
            import importlib
            ver = importlib.import_module(name).__version__
        except Exception:
            ver = "installed (version unknown)"
    print(f"{name:14s} {ver}")

try:
    import torch
    if torch.cuda.is_available():
        p = torch.cuda.get_device_properties(0)
        print(f"GPU: {p.name}, {p.total_memory / 2**30:.1f} GiB")
    elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
        print("GPU: Apple Silicon (MPS)")
    else:
        print("GPU: none found (CPU only)")
except ImportError:
    print("GPU: cannot tell, torch is not installed")
print("nvidia-smi:", shutil.which("nvidia-smi") or "not found")

Before installing anything, it prints each library as not installed and GPU: cannot tell, torch is not installed. After a correct install you should see versions, and either a GPU name with its memory, Apple Silicon (MPS), or none found.

#5. Common problems

#6. Habits that save runs

• Start with a tiny model and 50 examples to prove the pipeline end to end, then scale up.
• Save checkpoints and log the loss every few steps.
• Set the seed and write down the settings next to the result.
• Stop paid GPUs when you finish.

#Exercise

1. Run envcheck.py and write down what it reports.
2. You want to QLoRA an 8B model on a 12 GB card with sequence length 2048. Estimate the weight memory, and say what you would lower first if a run runs out of memory.

Answers: 2. About 3.7 GiB for the 4-bit weights plus under 1 GiB for the adapter and optimiser leaves several gigabytes for activations, so it should fit at a small batch. If it runs out of memory, reduce the batch size (and use gradient accumulation), then the sequence length.

Next: data basics: formats, chat templates and checks.