Run LLMs Locally with Ollama: Complete Guide (2026)

Run LLMs locally with Ollama: install it, pick a model that fits your RAM or GPU, use the REST API from code, tune context and fix the most common errors.

If you want to run LLMs locally with Ollama, you can go from zero to a working chat model in about five minutes. Ollama downloads open-weight models, runs them on your CPU or GPU, and gives you a simple command line plus a REST API on port 11434. No API keys, no per-token bills, and your data never leaves the machine.

I started using Ollama to prototype AI features for client projects without sending customer data to a third party. Today it is the first thing I install on a new dev box. In this guide I cover installation on Linux, macOS and Windows, choosing a model that actually fits your hardware, calling the API from code, customising models with a Modelfile, and the errors I see most often.

Everything here reflects Ollama as of October 2026. The project moves fast, so if a flag looks different on your version, run ollama --help and check the official docs.

Prerequisites

  • A 64-bit machine running Linux, macOS (Apple Silicon recommended) or Windows 10/11
  • At least 8 GB of RAM for small models (1B to 4B parameters); 16 GB or more is much more comfortable
  • Optional: an NVIDIA GPU with recent drivers, an AMD GPU supported by ROCm, or an Apple Silicon Mac
  • 10 to 50 GB of free disk space, depending on how many models you download
  • Basic terminal skills; curl is handy for testing the API

Step 1: Install Ollama

On macOS and Windows, download the installer from ollama.com and run it. It installs the background service and the ollama command.

On Linux, the official install script sets up a systemd service called ollama that listens on 127.0.0.1:11434.

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Check the version and that the server is running
ollama --version
curl http://127.0.0.1:11434
# -> "Ollama is running"

# Linux: manage the service
sudo systemctl status ollama
sudo journalctl -u ollama -f   # live logs, useful for GPU problems

Step 2: Pick a model that fits your hardware

The single most important rule for running LLMs locally: the model must fit in memory. If it fits entirely in GPU VRAM, it is fast. If it spills into system RAM, it still works but slows down a lot.

Ollama models are usually quantized to around 4 bits by default. A rough rule of thumb I use is that a 4-bit model needs a bit more than half its parameter count in gigabytes, plus extra for the context window. So a 7B-8B model wants roughly 5-6 GB, a 14B model around 9-10 GB, and a 30B+ model 20 GB or more.

Good starting points in the Ollama library as of October 2026 are the Qwen 3 family, Gemma 3, Llama 3.x and OpenAI's gpt-oss-20b. Always check the model page on ollama.com/library for the available tags and the license.

  • 8 GB RAM, no GPU: 1B-4B models such as llama3.2:3b, gemma3:4b or qwen3:4b
  • 16 GB RAM or an 8 GB GPU: 7B-8B models such as qwen3:8b or llama3.1:8b
  • 16-24 GB VRAM or a 32 GB Mac: 14B-20B models such as qwen3:14b or gpt-oss:20b
  • 48 GB+ VRAM or a 64 GB+ Mac: 30B-70B class models

Step 3: Pull and run your first model

# Download a model (only the first time)
ollama pull qwen3:8b

# Chat interactively (type /bye to exit)
ollama run qwen3:8b

# One-shot prompt from the shell
ollama run qwen3:8b "Explain Laravel queues in two sentences"

# Useful housekeeping commands
ollama list          # downloaded models
ollama ps            # models loaded in memory, and CPU/GPU split
ollama show qwen3:8b # parameters, template, license, context length
ollama rm qwen3:8b   # delete a model

Step 4: Use the Ollama REST API from your code

The real power is the HTTP API. Your Laravel, Node.js or Python app can call it exactly like a hosted LLM. The /api/chat endpoint takes a list of messages and streams JSON lines by default; set stream to false for a single response.

Ollama also exposes an OpenAI-compatible endpoint under /v1, so most OpenAI SDKs work by changing the base URL. That makes it easy to develop locally and switch providers later.

# Native chat API (non-streaming)
curl http://127.0.0.1:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [
    { "role": "system", "content": "You are a concise senior developer." },
    { "role": "user", "content": "What is an N+1 query?" }
  ],
  "stream": false,
  "options": { "temperature": 0.3, "num_ctx": 8192 }
}'

// Node.js 18+ with the OpenAI SDK pointed at Ollama
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:11434/v1",
  apiKey: "ollama", // required by the SDK, ignored by Ollama
});

const res = await client.chat.completions.create({
  model: "qwen3:8b",
  messages: [{ role: "user", content: "Write a regex for UK postcodes" }],
});

console.log(res.choices[0].message.content);

Step 5: Customise a model with a Modelfile

A Modelfile lets you bake a system prompt and default parameters into a named model. I use this to give each project its own assistant with a bigger context window, so I do not have to send the same options with every request.

# Modelfile
FROM qwen3:8b
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
SYSTEM """
You are a code reviewer for a Laravel 11 and React codebase.
Answer briefly, point out security issues first, and show fixed code.
"""

# Build and run it
ollama create laravel-reviewer -f Modelfile
ollama run laravel-reviewer

Useful environment variables

Ollama is configured with environment variables. On Linux, set them with sudo systemctl edit ollama and add lines under [Service], then restart the service. On macOS and Windows, set them as user environment variables and restart the Ollama app.

# sudo systemctl edit ollama
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"   # keep it local unless you add auth
Environment="OLLAMA_KEEP_ALIVE=30m"         # how long a model stays loaded
Environment="OLLAMA_NUM_PARALLEL=2"         # concurrent requests per model
Environment="OLLAMA_MODELS=/data/ollama"    # where models are stored

sudo systemctl daemon-reload
sudo systemctl restart ollama

Common errors and how I fix them

  • Responses are painfully slow: run ollama ps. If it shows a CPU/GPU split or 100% CPU, the model does not fit in VRAM. Use a smaller model or a smaller num_ctx.
  • GPU not detected on Linux: check nvidia-smi works first, then read journalctl -u ollama for CUDA or ROCm messages. Reinstalling after installing drivers usually fixes it.
  • connection refused on port 11434: the service is not running (systemctl start ollama) or you are calling it from a Docker container, where you need host.docker.internal instead of 127.0.0.1.
  • The model forgets earlier parts of a long document: the default context window is small. Raise num_ctx in options or in a Modelfile, keeping memory in mind.
  • Out of disk space: models live in ~/.ollama/models (or /usr/share/ollama on Linux service installs). Remove unused ones with ollama rm or move them with OLLAMA_MODELS.
  • Error pulling a tag: the tag name changed or does not exist. Check the exact tag on the model's library page.

Best practices and security

  • Never expose port 11434 to the internet without a reverse proxy and authentication. The API has no built-in auth, so anyone who can reach it can use your GPU.
  • Pin model tags in your code (qwen3:8b, not just qwen3) so an update does not silently change behaviour.
  • Read each model's license before shipping a commercial product. Licenses range from Apache 2.0 and MIT to custom community licenses with extra conditions.
  • Keep temperature low (0 to 0.3) for code and data extraction, higher for creative writing.
  • Treat model output as untrusted input: validate JSON, escape HTML and never run generated code or SQL without review.
  • Update Ollama regularly; new model architectures often need a newer runtime.

Conclusion

Running LLMs locally with Ollama is now genuinely practical for developers. Install it, choose a model that fits in your VRAM, talk to it through the REST or OpenAI-compatible API, and lock down the port. From there you can build chatbots, RAG systems and internal tools without sending a single byte to a third-party API.

In my next posts I build on this setup: a streaming chatbot in Laravel and a RAG chatbot over your own documents with Node.js.

Frequently asked questions

Can I run Ollama without a GPU?

Yes. Ollama runs on CPU only, but it is slower. Small models in the 1B to 4B range are usable on a modern laptop with 8 to 16 GB of RAM.

How much RAM do I need to run an LLM locally?

For a 4-bit quantized model, plan for a little more than half the parameter count in gigabytes plus room for the context: about 5-6 GB for an 8B model and 9-10 GB for a 14B model.

Is Ollama free for commercial use?

Ollama itself is open source under the MIT license. Each model has its own license, so check the model page before using it in a commercial product.

Does Ollama work with the OpenAI SDK?

Yes. Point the SDK's base URL to http://localhost:11434/v1 and pass any string as the API key. Chat completions and embeddings work for most use cases.

Where does Ollama store downloaded models?

By default in ~/.ollama/models on macOS and Windows, and under /usr/share/ollama/.ollama/models for the Linux service. Change it with the OLLAMA_MODELS environment variable.