Fine-Tune an LLM with LoRA on a Single GPU (QLoRA + PEFT)

Fine-tune an LLM with LoRA on a single GPU using Hugging Face PEFT, TRL and QLoRA: prepare data, train, evaluate, merge the adapter and run it in Ollama.

You can fine-tune an LLM with LoRA on a single consumer GPU. Instead of updating billions of weights, LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices that are a tiny fraction of its size. Combine it with 4-bit quantization (QLoRA) and a 7B-8B model fits comfortably on a 16-24 GB card.

Before you start, one honest piece of advice from my own projects: fine-tuning is rarely the first tool to reach for. If you want the model to know your documents, use RAG. If you want a format, try a better prompt with examples. Fine-tuning shines when you need a consistent style, a fixed output format, a narrow classification or extraction task, or a smaller model that behaves like a bigger one on one job.

In this guide I fine-tune an instruction model with Hugging Face Transformers, PEFT and TRL, then merge the adapter and run the result in Ollama. Library APIs evolve quickly, so I note where to check the current docs.

Prerequisites

  • An NVIDIA GPU with 12-24 GB VRAM (or a cloud GPU such as an L4, A10 or RTX 4090 instance)
  • Linux or WSL2 with recent NVIDIA drivers and CUDA working (nvidia-smi shows the GPU)
  • Python 3.10+
  • A Hugging Face account and access token if you use gated models
  • A few hundred to a few thousand high-quality training examples

Step 1: Install the libraries

python -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install transformers datasets peft trl accelerate bitsandbytes

huggingface-cli login   # only needed for gated models

Step 2: Prepare a chat-format dataset

TRL's SFTTrainer understands a messages column in the same role/content format used by chat APIs, and applies the model's chat template for you. I store data as JSONL, one conversation per line.

Data quality matters far more than quantity. Five hundred clean, consistent examples beat ten thousand scraped ones. Hold back around 10% for evaluation.

// data/train.jsonl (one JSON object per line)
{"messages": [{"role": "system", "content": "Extract invoice fields as JSON."}, {"role": "user", "content": "Invoice INV-204 from Acme Ltd, total GBP 1,250.00, due 2026-11-30"}, {"role": "assistant", "content": "{\"invoice_no\":\"INV-204\",\"supplier\":\"Acme Ltd\",\"currency\":\"GBP\",\"total\":1250.00,\"due_date\":\"2026-11-30\"}"}]}
{"messages": [{"role": "system", "content": "Extract invoice fields as JSON."}, {"role": "user", "content": "Bill #7781 - Northwind, amount due $89.90 by 5 Dec 2026"}, {"role": "assistant", "content": "{\"invoice_no\":\"7781\",\"supplier\":\"Northwind\",\"currency\":\"USD\",\"total\":89.90,\"due_date\":\"2026-12-05\"}"}]}

Step 3: The QLoRA training script

The script loads the base model in 4-bit NF4, attaches LoRA adapters to the attention and MLP projection layers, and trains with SFTTrainer. I picked a small Qwen instruct model so it trains quickly; swap in any causal LM whose license allows fine-tuning.

Key settings: r (the adapter rank) controls capacity, 16 is a good default; lora_alpha is usually 2x r; gradient checkpointing trades speed for memory; the learning rate for LoRA is typically around 1e-4 to 2e-4.

# train.py
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen2.5-1.5B-Instruct"   # try a 7B-8B model on a 24 GB GPU
OUT = "out/invoice-lora"

dataset = load_dataset("json", data_files={
    "train": "data/train.jsonl",
    "eval": "data/eval.jsonl",
})

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=bnb,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

args = SFTConfig(
    output_dir=OUT,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    bf16=True,
    gradient_checkpointing=True,
    max_length=1024,
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["eval"],
    processing_class=tokenizer,
    peft_config=lora,
)

trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model(OUT)          # saves only the small adapter
tokenizer.save_pretrained(OUT)

Step 4: Test the adapter

Before merging anything, load the base model plus the adapter and try prompts from your held-out set. Compare against the base model with the same prompts; if the fine-tuned version is not clearly better, fix the data before training longer.

# test.py
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

OUT = "out/invoice-lora"
model = AutoPeftModelForCausalLM.from_pretrained(OUT, device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(OUT)

messages = [
    {"role": "system", "content": "Extract invoice fields as JSON."},
    {"role": "user", "content": "Invoice 55-A from Globex, EUR 420.10, payable 1 Jan 2027"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Step 5: Merge the adapter and run it in Ollama

For deployment I merge the adapter into a full-precision copy of the base model, convert it to GGUF with llama.cpp, and import it into Ollama with a Modelfile. Merging into the 4-bit model is not recommended, so load the base model in bf16 for this step.

# merge.py
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "out/invoice-lora").merge_and_unload()
merged.save_pretrained("out/merged", safe_serialization=True)
AutoTokenizer.from_pretrained(BASE).save_pretrained("out/merged")

# Convert to GGUF and quantize with llama.cpp
python llama.cpp/convert_hf_to_gguf.py out/merged --outfile invoice-f16.gguf --outtype f16
./llama.cpp/build/bin/llama-quantize invoice-f16.gguf invoice-q4_k_m.gguf Q4_K_M

# Modelfile
FROM ./invoice-q4_k_m.gguf
PARAMETER temperature 0

# Import and run
ollama create invoice-extractor -f Modelfile
ollama run invoice-extractor "Invoice 9 from Initech, total 300 USD, due 2026-12-01"

Common errors

  • CUDA out of memory: lower per_device_train_batch_size to 1 or 2 and raise gradient_accumulation_steps, reduce max_length, keep gradient checkpointing on, or use a smaller base model.
  • bitsandbytes cannot find CUDA: your PyTorch, CUDA and bitsandbytes builds do not match. Check python -c "import torch; print(torch.cuda.is_available())" first.
  • TypeError about an unexpected argument in SFTConfig or SFTTrainer: TRL renamed some parameters between versions (for example tokenizer became processing_class). Check the docs for your installed version.
  • Loss drops to near zero but outputs are worse: overfitting. Use fewer epochs, more varied data, and watch eval loss.
  • The merged model ignores the chat format in Ollama: the GGUF has no template. Add a TEMPLATE to the Modelfile or rely on the template embedded during conversion, and test with ollama show.
  • bf16 not supported: older GPUs (pre-Ampere) need fp16=True instead of bf16=True.

Best practices

  • Start with a strong prompt and RAG baseline, and only fine-tune if it clearly falls short.
  • Build an evaluation set first and score base vs fine-tuned output on it; do not judge by a few hand-picked prompts.
  • Use the same system prompt and chat format in training and in production.
  • Remove personal data from training sets; models can memorise and repeat it.
  • Check the base model's license allows fine-tuning and distribution of derivatives.
  • Version datasets, adapters and hyperparameters together so you can reproduce a model.

Conclusion

Fine-tuning an LLM with LoRA on a single GPU is now a routine job: QLoRA keeps memory low, PEFT keeps the adapter small, and TRL handles the training loop and chat templates. Use it for narrow, repeatable tasks with good data, measure it against a baseline, and ship the merged model with Ollama or vLLM like any other open model.

Frequently asked questions

What is the difference between LoRA and QLoRA?

LoRA trains small adapter matrices while the base model is frozen. QLoRA does the same but loads the frozen base model in 4-bit precision, which cuts GPU memory use dramatically.

How much VRAM do I need to fine-tune a 7B model?

With QLoRA, a 7B-8B model can usually be fine-tuned on a 16-24 GB GPU with small batch sizes and gradient checkpointing. Full fine-tuning needs far more.

How many examples do I need for LoRA fine-tuning?

For narrow tasks like extraction or a consistent style, a few hundred to a few thousand high-quality examples is often enough. Quality and consistency matter more than volume.

Should I use fine-tuning or RAG?

Use RAG to give a model knowledge of your documents that changes over time. Use fine-tuning to change behaviour, format or style on a narrow task. Many production systems use both.