Hugging Face LoRA Fine-Tuning with PEFT

LoRA fine-tuning with Hugging Face PEFT lets you adapt a large language model by training tiny add-on matrices instead of all its weights. PEFT stands for parameter-efficient fine-tuning, and LoRA (low-rank adaptation) is its most popular method.

Instead of updating a 7-billion-parameter model, you might train only 10-40 million parameters. That means far less GPU memory, faster training and adapter files of a few megabytes instead of many gigabytes.

How LoRA Works

For selected weight matrices (usually the attention projections), LoRA freezes the original weight W and learns two thin matrices A and B. Their product B times A has the same shape as W but low rank r, often 8 or 16. During inference the model uses W plus a scaled B times A. Because r is small, A and B contain very few numbers.

Adding a LoRA Adapter

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto")

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

trainable params: 2,162,688 || all params: 496,195,456 || trainable%: 0.4359

Less than half a percent of the weights will be trained. The exact counts depend on the model and on which modules you target; target_modules="all-linear" is a convenient option that adapts every linear layer.

Training the Adapter

The PEFT model is a normal PyTorch module, so you can train it with Trainer. Many people use the TRL library's SFTTrainer, which adds chat-template formatting for instruction datasets:

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

data = load_dataset("json", data_files="my_chats.jsonl")["train"]  # rows with a "messages" list
trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    args=SFTConfig(output_dir="qwen-lora", num_train_epochs=1,
                   per_device_train_batch_size=4, learning_rate=2e-4),
)
trainer.train()

{'loss': 1.84, 'epoch': 0.25}
{'loss': 1.21, 'epoch': 0.5}
...
(sample output — losses vary)

LoRA usually uses a higher learning rate (around 1e-4 to 2e-4) than full fine-tuning because only the small adapter learns.

Saving, Loading and Merging

model.save_pretrained("qwen-lora/adapter")   # only a few MB

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto")
tuned = PeftModel.from_pretrained(base, "qwen-lora/adapter")

merged = tuned.merge_and_unload()     # fold the adapter into the weights
merged.save_pretrained("qwen-merged")

qwen-lora/adapter/adapter_config.json, adapter_model.safetensors written
qwen-merged/ written with full model weights

QLoRA and Practical Tips

• QLoRA loads the frozen base model in 4-bit (see the quantization lesson) and trains LoRA on top, fitting 7B-8B models on a single 16-24 GB GPU.
• Quality of your training data matters far more than the exact r value.
• Keep a held-out set of prompts and compare answers before and after tuning.
• Several adapters can share one base model and be switched at run time.

Frequently asked questions

What is LoRA in LLM fine-tuning?

Low-rank adaptation: the original weights are frozen and small low-rank matrices are trained and added to them, drastically reducing memory and storage needs.

What is the difference between LoRA and QLoRA?

QLoRA is LoRA applied to a base model that has been quantized to 4-bit, which lowers memory even further so larger models fit on smaller GPUs.

Should I merge a LoRA adapter?

Merge when you want a single standalone model for deployment with no extra latency. Keep it separate when you want to swap several adapters on one base model.