Hugging Face Quantization: bitsandbytes and GGUF

Quantization in Hugging Face stores model weights with fewer bits, for example 4 bits instead of 16, so a large language model uses far less memory and often runs faster. It is the main reason people can run 7B-70B parameter models on consumer GPUs and laptops.

How Much Memory Does Quantization Save?

  • 16-bit (bfloat16): about 2 bytes per parameter, so a 7B model is roughly 14 GB.
  • 8-bit: about 1 byte per parameter, roughly 7 GB.
  • 4-bit: about half a byte per parameter, roughly 4 GB plus some overhead.
  • You also need memory for activations and the KV cache, which grows with context length.

4-bit Loading with bitsandbytes

bitsandbytes quantizes weights on the fly while loading. It needs an NVIDIA GPU for full support (other backends are improving). NF4 is a 4-bit format designed for the normal distribution of neural network weights.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb, device_map="auto",
)
print(round(model.get_memory_footprint() / 1e9, 1), "GB")
5.5 GB   (sample — compared with about 15 GB in bfloat16)

compute_dtype controls the precision used for the actual math; the weights are stored in 4-bit and expanded on the fly.

8-bit Loading

bnb8 = BitsAndBytesConfig(load_in_8bit=True)
model8 = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct",
                                              quantization_config=bnb8, device_map="auto")
(model loaded with int8 linear layers)

What is GGUF?

GGUF is a single-file format used by llama.cpp and tools built on it, such as Ollama and LM Studio. GGUF files are pre-quantized, run well on CPUs and Apple Silicon, and come in variants named like Q4_K_M or Q8_0 (the number is the approximate bits per weight).

Many repositories on the Hub provide GGUF files. You can download one file with the hf CLI:

hf download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q4_k_m.gguf --local-dir ./models
./models/qwen2.5-0.5b-instruct-q4_k_m.gguf

Ollama can even pull a GGUF repository directly from the Hub with a command like ollama run hf.co/<user>/<repo>. transformers can also read GGUF files via the gguf_file argument, de-quantizing them for further training.

Choosing a Quantization Method

  • bitsandbytes: easiest in Python, great for QLoRA training on NVIDIA GPUs.
  • GGUF: best for local desktop apps, CPUs and Macs.
  • GPTQ, AWQ and similar: pre-quantized checkpoints optimized for fast GPU serving engines like vLLM.
  • Q4 levels usually keep most quality; go to Q5/Q6/Q8 if you notice errors in reasoning or code.

Frequently asked questions

Does quantization reduce model accuracy?

Slightly. 8-bit is usually almost identical to 16-bit, and good 4-bit methods lose only a little quality, which is often worth the large memory savings.

What does Q4_K_M mean in GGUF?

It is a llama.cpp quantization type: roughly 4 bits per weight using the k-quant scheme, in the medium size variant, a popular balance of size and quality.

Can I load a 7B model on an 8 GB GPU?

Usually yes in 4-bit, as the weights take around 4-5 GB. Leave room for the KV cache and keep context lengths moderate.