Dev.to AI 🤖 Ai 👁 0 📖 5 min read

Running an Agent Stack On-Prem: A Minimal Docker Compose Setup

Plenty of teams can't send their data to a hosted model, and plenty of others just don't want to. If you're in either group, the question is how small an on-prem agent stack can be while still being something you'd put i

Running an Agent Stack On-Prem: A Minimal Docker Compose Setup

Plenty of teams can't send their data to a hosted model, and plenty of others just don't want to. If you're in either group, the question is how small an on-prem agent stack can be while still being something you'd put in front of real users.

This is the smallest version I'd trust: a model server, a vector database and a thin gateway that handles keys and logging. The gateway is plain Node.js with no dependencies. I ran it against a fake upstream and the checks at the end pass. I haven't run the compose file itself, so check the image tags before you use it.

Running an Agent Stack On-Prem: A Minimal Docker Compose Setup

The three pieces

The model server runs the model. I use Ollama here because it's easy to start, but any server with an HTTP API fits the same slot. The vector database stores embeddings for retrieval, and Qdrant is a common self-hosted choice. The gateway is the part that makes it safe to share: it checks an API key, allows only the paths you expect and writes one log line per request.

Don't publish the model server's port. Everything should reach it through the gateway, so you have one place to authenticate and one place to look when something goes wrong.

The compose file

services:
  ollama:
    image: ollama/ollama
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant
    volumes:
      - qdrant:/qdrant/storage
    restart: unless-stopped

  gateway:
    build: ./gateway
    environment:
      GATEWAY_KEY: ${GATEWAY_KEY}
      OLLAMA_URL: http://ollama:11434
    ports:
      - "8080:8080"
    depends_on:
      - ollama
      - qdrant
    restart: unless-stopped

volumes:
  ollama:
  qdrant:

Only the gateway has a published port. Put the key in an .env file next to the compose file and keep that file out of version control.

The gateway

const http = require("node:http");

const UPSTREAM = process.env.OLLAMA_URL || "http://ollama:11434";
const KEY = process.env.GATEWAY_KEY;
const ALLOWED = new Set(["/api/chat", "/api/embeddings"]);

function log(entry) {
  console.log(JSON.stringify({ ts: new Date().toISOString(), ...entry }));
}

const server = http.createServer(async (req, res) => {
  const start = Date.now();
  const path = req.url.split("?")[0];
  const finish = (status, body, note) => {
    res.writeHead(status, { "content-type": "application/json" });
    res.end(JSON.stringify(body));
    log({ method: req.method, path, status, ms: Date.now() - start, note });
  };

  if (!KEY || req.headers["x-api-key"] !== KEY) return finish(401, { error: "unauthorized" }, "bad key");
  if (req.method !== "POST" || !ALLOWED.has(path)) return finish(404, { error: "not allowed" }, "blocked path");

  const chunks = [];
  for await (const c of req) chunks.push(c);
  try {
    const upstream = await fetch(UPSTREAM + path, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: Buffer.concat(chunks),
    });
    const text = await upstream.text();
    res.writeHead(upstream.status, { "content-type": "application/json" });
    res.end(text);
    log({ method: req.method, path, status: upstream.status, ms: Date.now() - start });
  } catch (err) {
    finish(502, { error: "upstream unavailable" }, String(err.message || err));
  }
});

module.exports = server;
if (require.main === module) server.listen(8080, () => log({ msg: "gateway listening", port: 8080 }));

The Dockerfile for it is four lines: start from a Node image, copy gateway.js, expose 8080 and run node gateway.js.

For a fuller picture of what a private rollout involves beyond this sketch, from sizing to upgrades and ongoing operations, see on-premise AI deployment.

Sizing and upgrades

Size the model server first, because it decides the hardware. A model that fits comfortably in memory answers quickly, and one that spills over crawls. Start with the smallest model that passes your own test questions, and move up only if it fails them. The vector database is usually the lighter part of the stack, and its disk use grows with the number of chunks you store.

Plan upgrades before you need them. Pin image tags instead of taking the latest version, so a restart doesn't pull a different model server by surprise. Back up the named volumes and rehearse a restore once. Then update one service at a time and run your test questions after each change.

Finally, decide who's on call. A stack like this has three moving parts and one gateway log, which is easy to look after, but only if one named person owns it.

Check that it behaves

This test starts a fake upstream, points the gateway at it and checks four things: a missing key is refused, a wrong key is refused, an unlisted path is blocked and a valid request gets through.

const assert = require("node:assert");
const http = require("node:http");
process.env.GATEWAY_KEY = "secret";

(async () => {
  const seen = [];
  const upstream = http.createServer((req, res) => {
    let b = ""; req.on("data", (c) => (b += c)); req.on("end", () => {
      seen.push(req.url);
      res.writeHead(200, { "content-type": "application/json" });
      res.end(JSON.stringify({ echo: JSON.parse(b) }));
    });
  });
  await new Promise((r) => upstream.listen(0, r));
  process.env.OLLAMA_URL = `http://127.0.0.1:${upstream.address().port}`;
  const gateway = require("./gateway.js");
  await new Promise((r) => gateway.listen(0, r));
  const base = `http://127.0.0.1:${gateway.address().port}`;
  const call = (path, headers = {}) =>
    fetch(base + path, { method: "POST", headers: { "content-type": "application/json", ...headers }, body: JSON.stringify({ q: 1 }) });

  assert.equal((await call("/api/chat")).status, 401);
  assert.equal((await call("/api/chat", { "x-api-key": "wrong" })).status, 401);
  assert.equal((await call("/api/delete", { "x-api-key": "secret" })).status, 404);
  const ok = await call("/api/chat", { "x-api-key": "secret" });
  assert.equal(ok.status, 200);
  assert.deepStrictEqual(seen, ["/api/chat"]);
  console.log("gateway test: all checks passed");
  gateway.close(); upstream.close();
})();

When on-prem is worth the effort

It pays off when data can't leave your network, when you want to control exactly when models change, or when per-request costs at high volume start to hurt. It costs you hardware, upgrades and someone on call for the stack.

If none of those apply, a hosted model is simpler. If one does, start with this small setup, measure, and add pieces only when you feel their absence. Logging is the first thing I'd add: the gateway already writes JSON lines, so shipping them to a log store is a small step.

Helping teams deploy AI agents on your own infrastructure is my day job at AI Hive, and the sketches that survive contact with production usually start this small. What would you put in front of the model first?

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.