Ollama Quantization

Ollama quantization is the reason big models fit on ordinary computers. Quantization stores each model weight with fewer bits, for example 4 bits instead of 16, cutting memory use by about four times with only a small loss in quality.

Common Quantization Levels

  • FP16 / BF16 - full precision, largest files, best quality.
  • Q8_0 - 8-bit, about half the size of FP16, nearly identical quality.
  • Q6_K / Q5_K_M - good middle ground.
  • Q4_K_M - 4-bit, the usual default; big savings, small quality drop.
  • Q3 / Q2 - very small, noticeably worse; only for tight hardware.

Estimating Memory

A quick rule of thumb: memory in GB is roughly parameters (in billions) times bits per weight divided by 8, plus overhead for the context cache.

8B model at 4-bit:  8 * 4 / 8  = ~4 GB  (+ ~1 GB overhead)
8B model at 8-bit:  8 * 8 / 8  = ~8 GB
8B model at 16-bit: 8 * 16 / 8 = ~16 GB
(approximate - actual sizes vary by architecture)

Choosing a Quantization Tag

Many models in the library offer tags for each quantization. Look at the "Tags" tab on the model page.

ollama pull llama3.1:8b-instruct-q8_0
ollama show llama3.1:8b-instruct-q8_0
  Model
    parameters      8.0B
    quantization    Q8_0

Rules of Thumb

  • A bigger model at Q4 usually beats a smaller model at Q8.
  • If a model fits at Q8 with room for context, use it for best quality.
  • Coding and maths suffer more from heavy quantization than casual chat.
  • You can quantize your own FP16 model with ollama create --quantize q4_K_M.

Frequently asked questions

What does Q4_K_M mean?

Q4 means about 4 bits per weight, K refers to the k-quant method, and M means the medium variant of that method.

Does quantization make models dumber?

Slightly. 8-bit is almost lossless, 4-bit is a small drop, and below 4-bit quality falls off faster.

What is GGUF?

GGUF is the file format used by llama.cpp and Ollama to store quantized models in a single file.