The 3 AM Gateway Leak: Why Your Next.js LLM Integration Needs Server Boundaries

It is 3 AM. Your on-call alert fires: someone posted your LLM gateway API key to a public GitHub issue. Not because of a breach. Because a developer fixed a missing browser credential by adding NEXT_PUBLIC_ to your .env, and the gateway client landed in a shared module that Client Components could import. The expensive mistake in an AI-enabled Next.js application is rarely model selection. It is treating an authenticated inference request like an ordinary frontend fetch.

The failure is architectural, not accidental. A gateway adapter becomes a shared utility. Someone adds a prefix. The application works. And 10,000 lines of minified JavaScript now ship your production secrets to every browser that downloads your app.

This is how you avoid that page.

The Boundary That Matters

The LLM gateway lives on the server side of Next.js. Always. The topology is trivial but unforgiving:

Browser → authenticated Server Action → server-only gateway adapter → provider

Gateway-side key encryption does not protect credentials accidentally shipped by your application. Upstream failover does not replace application authorization or request budgets. The trust boundary is yours alone.

I am evaluating tashfeenahmed/freellmapi as an external integrator for this architecture. It advertises 87 free providers, 1,313 model endpoints, routing, and encrypted key storage. Those are repository claims, not independently verified measurements. More importantly: its stated scope is "personal experimentation only." For a development playground or authenticated testing environment, that is exactly what you need. Assume the free tier has no production SLA.

The application boundary remains identical whether you route through b-lost, freellmapi, or Anthropic directly. Your responsibility is unchanged: credential isolation, session verification, and safe error handling.

Configuration and Implementation

This targets Next.js 15/16 App Router with an existing Auth.js integration. It requires zod and server-only. The example uses b-lost's relay endpoint (https://b-lost.com/v1) as a baseline; substitute your freellmapi instance's documented /v1 URL and credentials.

// .env.local — server-only; never prefix these with NEXT_PUBLIC_
// LLM_BASE_URL=https://b-lost.com/v1
// LLM_API_KEY=replace-with-your-gateway-key
// LLM_MODEL=replace-with-an-available-model-id

// lib/llm.ts
import "server-only";
import { z } from "zod";

const env = z.object({
  LLM_BASE_URL: z.string().url(),
  LLM_API_KEY: z.string().min(1),
  LLM_MODEL: z.string().min(1),
}).parse(process.env);

export const model = env.LLM_MODEL;

export async function gateway(path: string, body?: unknown) {
  const response = await fetch(
    `${env.LLM_BASE_URL.replace(/\/$/, "")}/${path}`,
    {
      method: body === undefined ? "GET" : "POST",
      headers: {
        Authorization: `Bearer ${env.LLM_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: body === undefined ? undefined : JSON.stringify(body),
      cache: "no-store",
      signal: AbortSignal.timeout(15_000),
    },
  );
  if (!response.ok) throw new Error("Gateway request failed");
  return response.json() as Promise<unknown>;
}

// app/playground/actions.ts
"use server";

import { z } from "zod";
import { auth } from "@/auth";
import { gateway, model } from "@/lib/llm";

const Input = z.object({
  prompt: z.string().trim().min(1).max(2000),
}).strict();

const Completion = z.object({
  choices: z.array(z.object({
    message: z.object({ content: z.string() }),
  })).min(1),
});

export type GenerateResult =
  | { ok: true; text: string }
  | {
      ok: false;
      code: "UNAUTHORIZED" | "INVALID_INPUT" | "UNAVAILABLE";
    };

export async function generate(input: unknown): Promise<GenerateResult> {
  try {
    const session = await auth();
    if (!session?.user?.id) {
      return { ok: false, code: "UNAUTHORIZED" };
    }

    const parsed = Input.safeParse(input);
    if (!parsed.success) return { ok: false, code: "INVALID_INPUT" };

    const result = Completion.parse(await gateway("chat/completions", {
      model,
      messages: [{ role: "user", content: parsed.data.prompt }],
      max_tokens: 256,
      stream: false,
    }));
    return { ok: true, text: result.choices[0].message.content };
  } catch {
    return { ok: false, code: "UNAVAILABLE" };
  }
}

// app/playground/page.tsx
import { Suspense } from "react";
import { z } from "zod";
import { auth } from "@/auth";
import { gateway } from "@/lib/llm";

function Skeleton({ label }: { label: string }) {
  return <p role="status" aria-busy="true">{label}...</p>;
}

async function Account() {
  const session = await auth();
  return <p>{session?.user?.id ? "Signed in" : "Sign in required"}</p>;
}

async function Models() {
  const session = await auth();
  if (!session?.user?.id) return null;
  try {
    const result = z.object({
      data: z.array(z.object({ id: z.string() })),
    }).parse(await gateway("models"));
    return <p>{result.data.length} models advertised</p>;
  } catch {
    return <p>Model discovery unavailable</p>;
  }
}

export default function Playground() {
  return (
    <main>
      <h1>Gateway playground</h1>
      <Suspense fallback={<Skeleton label="Loading account" />}>
        <Account />
      </Suspense>
      <Suspense fallback={<Skeleton label="Discovering models" />}>
        <Models />
      </Suspense>
    </main>
  );
}

Why This Matters at Scale

A Client Component can safely import generate because Next.js creates an action reference rather than bundling its implementation. The server-only import ensures accidental client imports fail at build time, not production.

Session verification happens inside the action because hiding a button is not authorization. Zod rejects oversized or malformed prompts before inference; the response schema prevents an incompatible provider response from becoming an unchecked application value. Errors never leak upstream bodies, credentials, or stack traces.

Separate Suspense boundaries for account and model discovery eliminate a rendering waterfall: the heading renders immediately while slower work proceeds independently. This removes one avoidable latency cliff without guaranteeing overall LCP improvement.

The Operational Debt You Are Deferring

Authentication is not a spending limit. Before moving beyond a controlled playground, add atomic per-user quota enforcement, deployment-level request limits, and request-ID-based observability without prompt logging.

The 15-second timeout bounds application waiting, not upstream billing. Automatic retries risk duplicating inference after ambiguous failures. Keep retry ownership in one layer; investigate the gateway's failure semantics before enabling application retries.

OpenAI-compatible gateways diverge on model identifiers, token-limit parameters, and response shapes. Integration tests comparing freellmapi against b-lost using identical prompts and compatible models—recording latency, failures, and output validity separately—are not optional before production routing decisions.

The Hardest Question

The useful production lesson is modest: keep trust decisions local, credentials server-only, and slow discovery out of critical rendering paths. But here is the question that haunts every LLM integration I have audited: when your gateway provider disappears or raises pricing 10x overnight, how do you safely rotate credentials and swap providers without invalidating every cached token or conversation ID in flight? Your Server Action boundary survives provider churn, but your data model probably does not. Have you thought through that operational nightmare yet?

nextjs #react #typescript #webdev