Ollama with Node.js

Using Ollama with Node.js usually means putting a small backend between your users and the Ollama server. The backend adds authentication, a fixed system prompt and rate limiting, and keeps Ollama private.

A Basic Express Endpoint

import express from 'express';
import ollama from 'ollama';

const app = express();
app.use(express.json());

app.post('/api/ask', async (req, res) => {
  try {
    const { question } = req.body;
    const r = await ollama.chat({
      model: 'llama3.2',
      messages: [
        { role: 'system', content: 'You are a helpful assistant for our docs site.' },
        { role: 'user', content: String(question).slice(0, 2000) },
      ],
    });
    res.json({ answer: r.message.content });
  } catch (err) {
    res.status(502).json({ error: 'Model unavailable' });
  }
});

app.listen(3000, () => console.log('Listening on 3000'));

Listening on 3000

POST /api/ask {"question":"What is Node.js?"}
-> {"answer":"Node.js is a JavaScript runtime that lets you run JavaScript on the server."}
(Sample response.)

Streaming to the Browser

Send chunks as they arrive so users see text appear immediately:

app.post('/api/stream', async (req, res) => {
  res.setHeader('Content-Type', 'text/plain; charset=utf-8');
  const stream = await ollama.chat({
    model: 'llama3.2',
    messages: [{ role: 'user', content: String(req.body.question) }],
    stream: true,
  });
  for await (const part of stream) res.write(part.message.content);
  res.end();
});

(the browser receives the answer word by word)

Production Tips

• Read the model name and Ollama host from environment variables.
• Limit input length and requests per user.
• Set OLLAMA_NUM_PARALLEL / queue requests; one GPU serves few users at once.
• Keep popular models warm with keep_alive to avoid cold-start delays.

Frequently asked questions

Can many users share one Ollama server?

Yes, but throughput is limited by the GPU. Ollama handles a few parallel requests and queues the rest.

Should the frontend call Ollama directly?

Usually not. Route through your backend so you control access, prompts and limits.

How do I deploy Ollama with my app?

Run Ollama as a service or Docker container on a GPU server and point your app at it via an environment variable.