Run a Capable LLM on Your Own Machine

Running a model locally sounds like a weekend project. It's actually ten minutes. This is the copy-paste version: pick a tool, pull a model, serve it over an OpenAI-compatible endpoint, and make your first tool call, with no hand-waving.

Treat this as a starting point for getting a model running, not a fine-tuning guide. The aim is a working endpoint in ten minutes; the deeper performance knobs (sampler settings, speculative decoding, GPU offload splits) are their own rabbit hole, and a separate post.

Why local, when hosted models are right there and genuinely good? Privacy is the obvious one: nothing leaves the machine. Then cost, because a long or repetitive job stops metering you by the token. And control, which matters more than it sounds. No rate limits, no refusal out of nowhere, and the version stays pinned instead of shifting under you one morning. You pay for all of it in quality and speed, and that price is real. The last section is about when it's worth paying.

First decision: which runner

Three tools cover almost everyone. Pick by how you work:

I'll use LM Studio's CLI (lms) as the main path, because it gives you a clean OpenAI-compatible server and stays out of your way on model management, then show the Ollama and llama.cpp equivalents.

Sizing: will it even fit?

Before pulling anything, do the memory math. A quantized model's file size is roughly its RAM/VRAM footprint, plus headroom for the context (KV cache).

Rule of thumb: model file size + ~20% ≤ your available RAM/VRAM.

• 8 GB → a 7-8B model at Q4 (≈4-5 GB).
• 16 GB → a 13-14B at Q4, or an 8B comfortably.
• 32 GB → a ~27-34B at Q4.
• 64 GB → a 70B at Q4 starts to be realistic.

Quantization (Q4/Q5/Q8) trades quality for size. Q4_K_M is the usual sweet spot; go Q8 only if you have the memory to spare and want the extra fidelity.

LM Studio (via the lms CLI)

Install LM Studio, then the CLI ships with it. List what you have, pull a model from the UI or CLI, and load it:

# what's available locally
lms ls

# load a model, with an explicit context length for headroom
lms load <model-name> --context-length 16384 --yes

# what's loaded right now
lms ps

# free the memory when you're done
lms unload <model-name>

LM Studio serves an OpenAI-compatible API (default http://localhost:1234/v1), and that compatibility is the whole point: anything that speaks the OpenAI API (your scripts, LiteLLM, an agent framework) points at the endpoint with a dummy key and works unchanged.

# confirm the server and see loaded models
curl -s http://localhost:1234/v1/models | jq '.data[].id'

A first chat completion:

curl -s http://localhost:1234/v1/chat/completions \
  -H "Authorization: Bearer lm-studio" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role":"user","content":"Say hi in five words."}]
  }' | jq -r '.choices[0].message.content'

The same thing in Ollama

# pull and run (first run downloads)
ollama run llama3.1:8b "Say hi in five words."

# serve the API (runs on :11434)
ollama serve

Ollama also exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, so the same curl as above works with the port swapped.

The same thing in llama.cpp

For headless servers or maximum control:

# build once (see the repo for your platform), then serve a GGUF file
llama-server -m ./models/your-model.Q4_K_M.gguf \
  -c 16384 --host 0.0.0.0 --port 8080

llama-server also speaks the OpenAI API (/v1/chat/completions). One thing to watch on long sessions: the KV cache grows with context and RSS climbs over time; if latency degrades, restart the server to reset it.

Tool calling (the part most local models fumble)

If you want the model to do things (call functions, drive tools), you need tool-calling, and local models vary wildly here. Send the OpenAI tools array and see if it responds with a structured tool_calls:

curl -s http://localhost:1234/v1/chat/completions \
  -H "Authorization: Bearer lm-studio" -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role":"user","content":"What time is it in UTC?"}],
    "tools": [{
      "type":"function",
      "function":{
        "name":"get_time",
        "description":"Return the current time for a timezone",
        "parameters":{"type":"object","properties":{"tz":{"type":"string"}}}
      }
    }],
    "tool_choice":"auto"
  }' | jq '.choices[0].message.tool_calls'

If you get a clean tool_calls array back, the model can drive tools. If you get prose describing the call instead, that model (or that quant) isn't reliable for agentic work; pick another, or keep the tool schema small and simple. This is the single biggest gotcha with local models, and it's worth its own article.

Choosing honestly

Local is not the default answer. Here's how I'd actually decide:

• Go local when the data cannot leave the machine, when the job is high-volume or repetitive, or when you need a pinned model that nothing rate-limits.
• Stay hosted when you need top-tier reasoning, reliability across long chains of steps, or you simply don't want to babysit memory and quantization.
• Most real setups land in between: local for the bulk, a hosted model for the handful of genuinely hard steps.

Takeaway

Pick a runner, do the memory math, load a model, hit the endpoint. That part really is ten minutes. The decisions that matter come after: how far you quantize, and whether the model can hold tool-calling, because that is what makes or breaks anything agentic you put on top of it.

Next: why local models fail at tool-calling, and how to design around it.