The Local LLM Production Trap: Why MoE Offloading, Agent Fleets, and Silent fetch() Timeouts Break at 3 a.m.

Originally published on tamiz.pro.

At 3:04 a.m. on a Tuesday, your self-hosted LLM stack stops answering. Not with a 500, not with a clean timeout — it just… stops. The pager fires. You SSH in and find a process chewing 100% CPU, a GPU sitting at 0% utilization, and a log file full of ECONNRESET errors. The agent fleet that was supposed to be idle is actually mid-retry-storm. The MoE model you offloaded to CPU RAM has thrashed its way into a swap loop. And the fetch() call that triggered it all — the one that has been hanging for 47 minutes — never had a timeout in the first place.

This is the local LLM production trap. It isn't one bug. It's three independent design decisions that are each reasonable in isolation, but which compound into a cascade failure at the worst possible moment. Let's dissect all three, see exactly how they interact, and then harden the stack against them.

• 1. The Promise of Local LLMs in Production
• 2. MoE Offloading: The Expert That Isn't There
• 3. Agent Fleets: Retry Storms and Feedback Loops
• 4. Silent fetch() Timeouts: The Unhandled Rejection
• 5. The 3 a.m. Convergence
• 6. Hardening the Stack
• 7. Frequently Asked Questions

1. The Promise of Local LLMs in Production

Self-hosting a large language model has become a legitimate production strategy. Teams adopt it for data residency, for per-token cost control, for latency predictability, or to escape rate limits on hosted APIs. The hardware is affordable — a single 24 GB GPU can run a 7B model comfortably and a 70B model with quantization and offloading. The tooling is mature: llama.cpp, vLLM, Ollama, SGLang. And the ecosystem is buzzing with agent frameworks that promise autonomous, multi-step reasoning.

The trap is that these systems pass the daytime test. During business hours, you run 10 concurrent requests, p95 latency is 800 ms, and everything looks great. You deploy. You go home. And then the night shift — cron jobs, batch ETL, backups, model reloads, and the agent fleet's scheduled maintenance — starts firing.

Production is not about average latency. It's about tail latency, and tail latency is where local LLM stacks die. Three failure modes account for the vast majority of 3 a.m. incidents:

1. MoE offloading thrash — the router evicts and reloads experts faster than the PCIe bus can transfer them.
2. Agent fleet retry storms — correlated retries synchronize into a thundering herd.
3. Silent fetch() timeouts — requests hang forever because nobody set a timeout, and the server keeps burning compute.

Each one alone is survivable. Together, they form a feedback loop that's nearly impossible to break without a restart. Let's look at each in detail.

2. MoE Offloading: The Expert That Isn't There

What MoE Actually Is

A Mixture of Experts (MoE) model replaces each feed-forward layer with a set of smaller expert networks and a router. The router computes a probability distribution over experts and selects the top-k for each token. In an 8x7B model, the router picks 2 of 8 experts per token. This gives you dense-model quality with sparse-compute efficiency — in theory.

In practice, MoE models are enormous. Mixtral 8x7B is about 47 GB in FP16. An 8x22B model is ~140 GB. Even a single 24 GB GPU can't hold the whole thing, so you offload.

The Offloading Decision

Offloading means keeping the attention layers, embeddings, and router on the GPU, and moving the expert FFN weights to CPU RAM (or to a second, slower GPU). llama.cpp exposes this via --n-cpu-moe and --cpu-moe flags. vLLM has --cpu-offload-gb. The idea is sound: most prompts route through a small subset of experts, so you can keep the "hot" experts resident and page in the rest.

# llama.cpp: offload experts to CPU, keep attention on GPU
./llama-server \
  --model mixtral-8x7b-instruct.Q4_K_M.gguf \
  --n-gpu-layers 20 \
  --n-cpu-moe 4 \
  --cpu-moe \
  --parallel 4 \
  --ctx-size 8192

The problem is that expert routing is dynamic and data-dependent. You cannot predict which experts a prompt will hit. A prompt about "Rust async runtime internals" might route heavily to expert 3, while a prompt about "Medieval French poetry" routes to expert 6. When a batch of requests arrives, the router can oscillate, causing the CPU to page in and out of the same experts repeatedly.

Expert Thrash

This is the "expert thrash" phenomenon. The router picks experts, the scheduler loads them from CPU to GPU, the next token picks different experts, the scheduler evicts the previous ones. Each eviction is a PCIe transfer. PCIe Gen4 delivers about 32 GB/s — but that's theoretical. Real-world, with small transfer sizes and OS overhead, you're lucky to get 10 GB/s. A 7B expert is about 14 GB in FP16. Loading one expert takes over a second. And you're doing it per token, per request, per layer.

The result is a latency cliff. A request that should take 200 ms takes 12 seconds. And because the CPU is now the bottleneck, the GPU sits idle, waiting for weights that will be evicted before they're used.

KV Cache Contention

The second half of the MoE problem is memory. The KV cache — the attention keys and values for every token in every active context — grows linearly with context length and multiplicatively with batch size. A 32k context with 8 concurrent requests can eat 10+ GB of GPU memory. When you've offloaded experts to CPU to make room, the KV cache and the offloaded expert weights are competing for the same limited GPU memory. When the KV cache grows, the GPU either OOMs or starts evicting experts. When it evicts experts, latency spikes. When latency spikes, requests pile up, which grows the KV cache further.

This is a positive feedback loop, and it's the engine that drives the 3 a.m. failure.

3. Agent Fleets: Retry Storms and Feedback Loops

Agents Are Request Multipliers

An agent isn't one LLM call. It's a loop: think → call a tool → observe the result → think again. A single agent turn might be 3–10 LLM calls, each with a growing context. An "agent fleet" is N agents running concurrently. If you have 20 agents and each makes 5 calls, that's 100 LLM requests — but they're not independent. They're correlated. They all start at the same time (a cron trigger), they all hit the same model, and they all fail for the same reason.

The Retry Storm

When an agent's tool call times out, the framework retries. The retry is supposed to use exponential backoff with jitter. But here's the catch: backoff only works if the server is available. When the server is saturated — because of MoE thrash or KV cache pressure — every retry lands on a saturated server. The backoff delays the request, but it doesn't reduce the load. And because all agents in the fleet are retrying with the same backoff schedule, they synchronize. The "full jitter" that's supposed to desynchronize them only works if the random seed is actually random — and if the retry count is bounded.

// The classic retry loop that storms under load
async function withRetry(fn: () => Promise<any>, maxRetries = 5) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      // No jitter, no circuit breaker, no backoff cap
      await sleep(1000 * Math.pow(2, attempt)); // 1s, 2s, 4s, 8s, 16s
    }
  }
  throw new Error('exhausted');
}

Notice what happens: at attempt 3, the sleep is 8 seconds. If 20 agents all hit this, they all wake up at roughly the same time and all fire at once. The server, already struggling, gets a burst of 20 requests. It can't handle them. They all time out. They all retry. The storm continues.

Token Budget Exhaustion

There's a subtler problem. Each retry doesn't just re-send the original prompt. In an agent loop, the context grows with each turn. The retry might include the full conversation history — all tool outputs, all observations. So retry #4 might be 8,000 tokens when retry #1 was 2,000. This grows the KV cache, which pressures GPU memory, which triggers more expert eviction, which slows everything down.

Deadlock

The nastiest failure is a fleet-level deadlock. Agent A calls a tool that requires Agent B's output. Agent B calls a tool that requires the LLM server. The LLM server is waiting for memory. Agent A is waiting for Agent B. Agent B is waiting for the LLM. Nothing is making progress. The only thing that breaks the deadlock is a global timeout — which, if you didn't set one, never comes.

4. Silent fetch() Timeouts: The Unhandled Rejection

fetch() Has No Timeout

The fetch() API in Node.js, Bun, Deno, and browsers has no default timeout. A fetch() call can hang forever. This is by design — the Fetch spec doesn't include a timeout. The responsibility is on the caller. Most developers know this, but in practice, the codebase is full of bare await fetch(url) calls that assume the server will respond.

// The silent killer: no timeout, no abort, no response.ok check
async function callLLM(prompt: string) {
  const res = await fetch('http://localhost:8080/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'mixtral', messages: [{ role: 'user', content: prompt }] }),
  });
  // If res is 502 or 503, this still tries to parse it
  const data = await res.json();
  return data.choices[0].message.content;
}

The Three Silent Failures

There are three distinct ways this bites you:

1. The hang. The request is sent, the server is slow, the client waits. No timeout means it waits forever. The connection stays open, the server keeps processing, the client's event loop is blocked on a promise that never settles.

2. The non-2xx that isn't checked. The server returns a 503. The code doesn't check res.ok. It tries to parse the error body as JSON, gets undefined, and returns undefined as if it were a valid completion. The agent gets undefined, treats it as a tool result, and continues with garbage.

3. The streaming stall. The server starts streaming a response, sends a few chunks, then stalls — because the MoE model is thrashing or the GPU is OOMing. The client is reading a stream that never ends. No timeout, no AbortController, no readableStream close. The request hangs mid-body.

Unhandled Rejections

Even when you do have a timeout, the failure mode shifts. If you use Promise.race with a timeout but don't abort the underlying fetch, the request keeps running on the server. The client moves on, but the server is still burning compute. And if you don't catch the rejection, you get an unhandled promise rejection — which in Node.js can crash the process (depending on the unhandledRejection handler).

The right pattern is AbortController:

async function callLLM(prompt: string, timeoutMs = 30_000) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);
  try {
    const res = await fetch('http://localhost:8080/v1/chat/completions', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ model: 'mixtral', messages: [{ role: 'user', content: prompt }] }),
      signal: controller.signal,
    });
    if (!res.ok) {
      throw new Error(`LLM returned ${res.status}: ${await res.text()}`);
    }
    const data = await res.json();
    return data.choices[0].message.content;
  } catch (err) {
    if (err.name === 'AbortError') {
      throw new Error('LLM request timed out');
    }
    throw err;
  } finally {
    clearTimeout(timer);
  }
}

This is better, but it only protects the client. The server is still processing the request. The AbortController sends a DOMException on the client side, but the HTTP connection is closed — and depending on the server, the LLM inference continues. The KV cache entry isn't freed. The compute isn't stopped. You've saved the client, but the server is still in the trap.

5. The 3 a.m. Convergence

Now let's put it together. It's 3:00 a.m. Three things happen at once:

1. A nightly batch job starts. It loads a 32k-token context into the LLM to summarize a day's logs. This single request grows the KV cache by several GB.

2. The agent fleet's scheduled maintenance fires. 20 agents wake up, each with a 4-turn task. They all start issuing requests at the same time.

3. The MoE model is already running with 4 experts offloaded to CPU. The batch job's 32k context pushes the KV cache over the memory budget, forcing the scheduler to evict more experts.

Here's the cascade:

• The batch job's request routes through a cold expert. The scheduler pages it in from CPU. Latency spikes to 8 seconds.
• The agent fleet's 20 requests hit the same cold expert. They queue. Each one waits.
• The agent fleet's fetch() calls have no timeout. They wait. And wait. And wait.
• After 30 seconds, the agents' internal retry logic kicks in. They retry. The retries land on the same saturated server. They queue behind the batch job.
• The KV cache grows. The GPU OOMs. The server process crashes, or the OOM killer takes it.
• All in-flight requests fail. The agents see ECONNRESET. They retry — but now the server is restarting, so they get ECONNREFUSED. They retry again.
• The server comes back, reloads the model, re-warms the experts. The agents' retries hit it. It's cold. Latency is high. They time out. They retry.
• The batch job's supervisor sees the failure, restarts the job. It fires again. The cycle repeats.

This is the 3 a.m. convergence. None of these is a "bug" in isolation. The MoE offloading is a reasonable memory optimization. The agent fleet is a reasonable way to parallelize work. The bare fetch() is a reasonable assumption that the server will respond. But together, they form a system that is guaranteed to fail under a load pattern that is guaranteed to occur at night.

6. Hardening the Stack

1. Pin Your Experts, Don't Let Them Thrash

The first fix is to stop treating expert routing as unpredictable. Measure your workload. Log which experts are hit. Then pin the hot experts to the GPU and offload only the cold ones. Better yet, don't offload at all if you can avoid it — use a quantized model that fits entirely in VRAM. A Q4_K_M Mixtral is about 26 GB. A 24 GB GPU can't fit it, but a 2×24 GB setup can. Or use a smaller MoE.

If you must offload, reserve a dedicated memory budget for the KV cache and never let it exceed it. vLLM lets you do this with --kv-cache-memory-fraction and --cpu-offload-gb. Set them so that the KV cache and the offloaded experts never compete.

2. Add Timeouts Everywhere

Every fetch() call gets an AbortController with a timeout. Every streaming response gets a readableStream timeout. Every res.ok check is mandatory. This is non-negotiable. The code above is the template.

3. Fix the Retry Logic

Retries need three things: full jitter, a circuit breaker, and a request budget. Full jitter means sleep(random(0, backoff)) instead of sleep(backoff). A circuit breaker trips after N consecutive failures and refuses all requests for a cooldown period. A request budget caps the total number of in-flight requests across the fleet.

class CircuitBreaker {
  private failures = 0;
  private openedAt = 0;
  constructor(private threshold = 5, private cooldownMs = 30_000) {}
  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.openedAt && Date.now() - this.openedAt < this.cooldownMs) {
      throw new Error('Circuit open');
    }
    try {
      const result = await fn();
      this.failures = 0;
      this.openedAt = 0;
      return result;
    } catch (err) {
      this.failures++;
      if (this.failures >= this.threshold) {
        this.openedAt = Date.now();
      }
      throw err;
    }
  }
}

async function withRetry(fn: () => Promise<any>, maxRetries = 4) {
  const breaker = new CircuitBreaker();
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await breaker.call(fn);
    } catch (err) {
      const backoff = Math.random() * Math.min(1000 * Math.pow(2, attempt), 10_000);
      await sleep(backoff); // full jitter
    }
  }
  throw new Error('exhausted');
}

4. Rate-Limit the Fleet

Use a semaphore or a token bucket to cap concurrent requests. If you have 20 agents, cap the LLM at 4 concurrent requests. This prevents the thundering herd. The agents queue, but the queue is bounded, and the requests are spread out.

class Semaphore {
  private queue: (() => void)[] = [];
  private available: number;
  constructor(private max: number) { this.available = max; }
  async acquire(): Promise<void> {
    if (this.available > 0) { this.available--; return; }
    return new Promise((resolve) => this.queue.push(resolve));
  }
  release(): void {
    const next = this.queue.shift();
    if (next) { next(); } else { this.available++; }
  }
}

const llmSemaphore = new Semaphore(4);

async function guardedLLMCall(prompt: string) {
  await llmSemaphore.acquire();
  try {
    return await callLLM(prompt);
  } finally {
    llmSemaphore.release();
  }
}

5. Separate Batch from Interactive

The batch job and the agent fleet should never share the same server. Run two instances: one for interactive traffic with a small context and low concurrency, one for batch with a large context and high concurrency. Or use different model shards. This is the single most effective mitigation — it eliminates the cross-contamination that causes the cascade.

6. Pre-Warm and Cache

Before the night shift starts, pre-warm the experts. Cache the router decisions for common prompts. And for the batch job, don't send a 32k context in one shot — chunk it. A 32k context is a memory bomb. Chunking it into 4k pieces and summarizing incrementally keeps the KV cache small.

7. Monitor the Right Metrics

Don't just monitor p50 latency. Monitor p99, expert eviction rate, KV cache utilization, in-flight request count, and retry rate. The 3 a.m. failure shows up in these metrics before it shows up in the pager. Set alerts on expert eviction rate and KV cache utilization.

7. Frequently Asked Questions

Why do these failures happen specifically at 3 a.m.?

Because that's when scheduled jobs run. Cron, batch ETL, backups, nightly model reloads — all of these are scheduled for the quiet hours. They coincide with the agent fleet's maintenance window. During the day, the load is spread out and the system has headroom. At night, the load is bursty and correlated, which is exactly the pattern that triggers the cascade.

Is MoE offloading ever safe in production?

Yes, but only with strict memory budgeting and expert pinning. If you can measure your expert routing and pin the hot experts to GPU, offloading the cold ones is safe. The danger is when you let the router freely evict and reload experts under memory pressure. Always set a hard KV cache budget and never let it compete with offloaded weights.

Should I just use a hosted API instead?

If you can tolerate the latency and cost, a hosted API with a 99.9% uptime SLA is strictly more reliable than a self-hosted stack. The local LLM is worth it when you need data residency, custom tokenization, or very high throughput at low marginal cost. But you're trading reliability for control. If you go local, you must implement the hardening in this article — it's not optional. For more on production LLM infrastructure patterns, see Tamiz's Insights.