Stop Calling Your LLM Directly: Build a Small AI Gateway in Node.js

Timeouts, retries, error handling and provider abstraction for more reliable AI-powered applications.

You've integrated an LLM into your Node.js application. The API works, responses look good, and the feature is ready for testing.

Then something goes wrong.

The model provider takes 30 seconds to respond. Another request returns a 429 rate-limit error. A temporary provider outage breaks your application endpoint. Meanwhile, multiple parts of your codebase are implementing their own model calls, each with slightly different error handling.

This is where a simple AI integration starts becoming an architectural problem.

A common mistake is allowing every application module to communicate directly with an LLM provider.

A better approach is to introduce a small AI gateway that manages model communication centrally.

In this tutorial, we'll build a lightweight LLM gateway using TypeScript and Node.js. We'll cover provider abstraction, request timeouts, retry handling and failure management without introducing unnecessary infrastructure.

1. Why Do We Need an AI Gateway?

Consider an application that has three AI features:

• Document summarization.
• AI-powered text suggestions.
• Message translation.

Without a shared integration layer, the implementation might look like this:

Document Service
      |
      v
   LLM API

Suggestion Service
      |
      v
   LLM API

Translation Service
      |
      v
   LLM API

Each service handles authentication, request formatting, timeouts, errors and provider configuration independently.

Initially, this may seem manageable.

But imagine switching model providers or introducing a private model endpoint.

You now need to update multiple parts of your application.

Instead, centralize model communication:

Document Service ----+
Suggestion Service --+----> AI Gateway
Translation Service -+          |
                                 v
                         Provider Adapter
                                 |
                        +--------+--------+
                        |                 |
                        v                 v
                    Cloud LLM        Private LLM

The gateway doesn't need to contain every AI business capability. Its primary responsibility is to manage communication between the application and model providers.

This separation makes the application easier to maintain and extend.

2. Define a Common Provider Interface

Let's start with a simple TypeScript contract.

Create a file named llm-provider.ts.

export interface LLMRequest {
  systemInstruction: string;
  prompt: string;
  maxOutputTokens?: number;
  temperature?: number;
}

export interface LLMResponse {
  text: string;
  model: string;
  inputTokens?: number;
  outputTokens?: number;
}

export interface LLMProvider {
  generate(
    request: LLMRequest,
    signal: AbortSignal
  ): Promise<LLMResponse>;
}

The important part is the LLMProvider interface.

Our application doesn't need to know whether the implementation uses a hosted model, a private endpoint or a self-hosted model.

It only needs a predictable contract.

Each provider adapter can implement the same interface while translating requests and responses into the format expected by its underlying SDK or API.

This is an example of the adapter pattern applied to AI integrations.

3. Handle Provider Errors Properly

Not every failure should trigger a retry.

For example:

A rate-limit error requires different handling from an invalid API key.

Let's create a custom provider error.

export class LLMProviderError extends Error {
  constructor(
    message: string,
    public readonly options: {
      status?: number;
      retryAfterMs?: number;
      retryable?: boolean;
    } = {}
  ) {
    super(message);
    this.name = "LLMProviderError";
  }

  get isRetryable(): boolean {
    if (this.options.retryable !== undefined) {
      return this.options.retryable;
    }

    const status = this.options.status;

    return (
      status === 429 ||
      (status !== undefined &&
        status >= 500 &&
        status < 600)
    );
  }
}

export class LLMTimeoutError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "LLMTimeoutError";
  }
}

This allows the gateway to distinguish temporary provider problems from permanent request failures.

Your provider adapter should convert HTTP errors and network failures into these application-level errors.

For example, a failed network request can be represented as:

throw new LLMProviderError(
  "Temporary network failure",
  { retryable: true }
);

A provider should also extract the Retry-After header when available and pass its value to the gateway.

4. Implement Request Timeouts

One of the biggest mistakes in external API integration is allowing requests to wait indefinitely.

An AI request might become slow because of provider congestion, network issues or a large input.

Use AbortController to cancel requests after a defined timeout.

Node.js supports AbortController for cancellation of asynchronous operations. See the official Node.js documentation.

Here's a reusable timeout helper:

function withTimeout<T>(
  operation: (signal: AbortSignal) => Promise<T>,
  timeoutMs: number
): Promise<T> {
  const controller = new AbortController();

  let timer: ReturnType<typeof setTimeout> | undefined;

  const timeoutPromise = new Promise<never>(
    (_, reject) => {
      timer = setTimeout(() => {
        const error = new LLMTimeoutError(
          `Request timed out after ${timeoutMs}ms`
        );

        controller.abort(error);
        reject(error);
      }, timeoutMs);
    }
  );

  const requestPromise = Promise.resolve().then(
    () => operation(controller.signal)
  );

  return Promise.race([
    requestPromise,
    timeoutPromise
  ]).finally(() => {
    if (timer) {
      clearTimeout(timer);
    }
  });
}

There is an important detail here.

Promise.race() rejects when the timeout expires, but it doesn't automatically stop the original operation.

That's why we also abort the controller.

The provider adapter must pass the supplied signal into its underlying HTTP request or SDK. Otherwise, the operation may continue consuming resources after the gateway has timed out.

5. Add Retry Logic With Exponential Backoff

A temporary failure doesn't always mean the application should immediately return an error to the user.

For example, a provider might return HTTP 503 because of temporary server congestion.

A limited retry can improve reliability.

However, retrying immediately is not ideal. It can increase pressure on an already overloaded service.

Exponential backoff introduces increasing delays between attempts.

A basic implementation looks like this:

const sleep = (ms: number) =>
  new Promise<void>((resolve) => {
    setTimeout(resolve, ms);
  });

function isRetryable(error: unknown): boolean {
  return (
    error instanceof LLMTimeoutError ||
    (
      error instanceof LLMProviderError &&
      error.isRetryable
    )
  );
}

We'll use full jitter to avoid multiple requests retrying at exactly the same time.

The retry delay increases with every attempt but stays within a configured limit.

6. Put Everything Together in an LLM Gateway

Now let's create the main gateway.

Create llm-gateway.ts.

import {
  LLMProvider,
  LLMRequest,
  LLMResponse
} from "./llm-provider";

import {
  LLMProviderError,
  LLMTimeoutError
} from "./llm-errors";

type GatewayConfig = {
  timeoutMs: number;
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
};

const defaultConfig: GatewayConfig = {
  timeoutMs: 15000,
  maxAttempts: 3,
  baseDelayMs: 300,
  maxDelayMs: 4000
};

const sleep = (ms: number) =>
  new Promise<void>((resolve) => {
    setTimeout(resolve, ms);
  });

function isRetryable(error: unknown): boolean {
  return (
    error instanceof LLMTimeoutError ||
    (
      error instanceof LLMProviderError &&
      error.isRetryable
    )
  );
}

function withTimeout<T>(
  operation: (signal: AbortSignal) => Promise<T>,
  timeoutMs: number
): Promise<T> {
  const controller = new AbortController();

  let timer: ReturnType<typeof setTimeout> | undefined;

  const timeoutPromise = new Promise<never>(
    (_, reject) => {
      timer = setTimeout(() => {
        const error = new LLMTimeoutError(
          `Request timed out after ${timeoutMs}ms`
        );

        controller.abort(error);
        reject(error);
      }, timeoutMs);
    }
  );

  const requestPromise = Promise.resolve().then(
    () => operation(controller.signal)
  );

  return Promise.race([
    requestPromise,
    timeoutPromise
  ]).finally(() => {
    if (timer) clearTimeout(timer);
  });
}

export class LLMGateway {
  private readonly config: GatewayConfig;

  constructor(
    private readonly provider: LLMProvider,
    config: Partial<GatewayConfig> = {}
  ) {
    this.config = {
      ...defaultConfig,
      ...config
    };

    if (
      this.config.maxAttempts < 1 ||
      this.config.timeoutMs <= 0 ||
      this.config.baseDelayMs < 0 ||
      this.config.maxDelayMs < 0
    ) {
      throw new Error("Invalid LLM gateway configuration");
    }
  }

  async generate(
    request: LLMRequest
  ): Promise<LLMResponse> {

    if (!request.prompt.trim()) {
      throw new Error("Prompt cannot be empty");
    }

    for (
      let attempt = 1;
      attempt <= this.config.maxAttempts;
      attempt++
    ) {
      try {
        return await withTimeout(
          (signal) =>
            this.provider.generate(request, signal),
          this.config.timeoutMs
        );

      } catch (error) {

        if (
          attempt === this.config.maxAttempts ||
          !isRetryable(error)
        ) {
          throw error;
        }

        const exponentialLimit = Math.min(
          this.config.maxDelayMs,
          this.config.baseDelayMs *
            Math.pow(2, attempt - 1)
        );

        const jitter = Math.random() * exponentialLimit;

        const retryAfter =
          error instanceof LLMProviderError
            ? error.options.retryAfterMs ?? 0
            : 0;

        const delay = Math.min(
          this.config.maxDelayMs,
          Math.max(jitter, retryAfter)
        );

        await sleep(delay);
      }
    }

    throw new Error("LLM request failed");
  }
}

Now the application has one place to manage timeouts, retry limits and provider errors.

The implementation intentionally keeps provider-specific request formatting outside the gateway.

Also, make sure your provider adapter respects the cancellation signal. The timeout helper cannot forcibly terminate an SDK operation that ignores it.

7. Connect the Gateway to Your Application

You can now create a provider implementation and inject it into the gateway.

For example:

const gateway = new LLMGateway(
  yourProviderAdapter,
  {
    timeoutMs: 12000,
    maxAttempts: 3,
    baseDelayMs: 300,
    maxDelayMs: 3000
  }
);

Use the gateway from your application service:

const result = await gateway.generate({
  systemInstruction:
    "You are a helpful document summarization assistant.",

  prompt:
    "Summarize the following document: " + documentText,

  temperature: 0.2,
  maxOutputTokens: 800
});

console.log(result.text);

The application now depends on the gateway rather than a particular model SDK.

To change the provider, implement another adapter and inject it into the same gateway.

The business logic doesn't need to change.

8. What About Cost, Security and Production Monitoring?

Timeouts and retries improve request handling, but a production AI integration needs additional controls.

Cost management

Track input and output tokens, set application-level usage limits and establish quotas for individual users or tenants.

Remember that retrying a request can generate additional provider charges. Configure idempotency where supported, and avoid blindly retrying workflows that perform external actions.

Security

Never expose API keys in frontend code. Keep provider credentials on the server.

Validate access permissions before retrieving private application data or passing information to a model.

Also, avoid logging complete prompts and responses by default. They may contain sensitive customer information.

Monitoring

Track at least:

• Request latency.
• Provider error rates.
• Timeout frequency.
• Retry counts.
• Input and output token usage.
• Model-specific costs.
• Successful response rates.

These metrics help distinguish application problems from provider issues.

Input validation

Check request size and output constraints before invoking the model. Character limits can provide a basic guard, but token-aware limits are preferable because character counts do not accurately represent token consumption.

Circuit breakers

If a provider continues failing, a circuit breaker can temporarily stop new requests to that provider instead of allowing every application request to repeat the same failure.

For systems supporting multiple providers, a controlled fallback may also be useful.

9. Important: Don't Treat Every AI Operation the Same Way

A document summary, a translation request and an AI agent performing actions have different reliability requirements.

A short translation might be handled synchronously.

A large document summary may need background processing through a queue.

An AI agent that updates customer records should use a more carefully controlled execution model, with separate authorization and approval checks.

A request retry should never automatically repeat an irreversible business action.

In other words, the gateway should manage model communication, while the application remains responsible for business rules and execution permissions.

Final Thoughts

Integrating an LLM into a Node.js application is easy compared with operating that integration reliably.

Timeouts, retries, provider abstraction and error classification might look like small implementation details, but they make a significant difference when AI becomes part of a production application.

You don't need a complex microservice architecture to get started.

A small, well-designed gateway is often enough to centralize model communication and create room for future changes.

Start with a clean provider interface, introduce bounded retries, enforce request timeouts and measure what happens in production.

As the application grows, you can extend the same foundation with caching, rate limiting, observability and provider routing.

The goal isn't to build another layer of unnecessary abstraction. It's to make sure an unreliable external dependency doesn't make your entire application unreliable.