Before You Add an LLM, Build a Baseline

When a team says “let’s add AI,” it is tempting to start by comparing models. Start with the user’s task instead.

A baseline is the simplest version of a solution that helps with the task before introducing a model. It gives you a clear point of comparison and often shows that search, rules, or a better workflow can solve part of the problem.

Example: routing support tickets

Suppose a support team receives billing, account-access, and technical tickets. The goal is not “use an LLM.” The goal is to get each ticket to the right queue quickly, while sending uncertain cases to a person.

Write the task in one sentence:

Given a support ticket, suggest a queue and explain why; send uncertain cases to a human reviewer.

Start with clear rules for obvious cases, such as invoices going to billing and password-reset requests going to account access. Keep the suggestion visible for a reviewer. Save a representative test set that includes incomplete, ambiguous, and out-of-scope tickets.

Then compare a model

Run the model on the same examples and compare correctness by category, unsafe mistakes, review rate, response time, and cost per ticket. A model is useful only if the measured gain is worth its extra cost and operational work. Keep rules where they work well.

Design the failure path

Treat model output as a suggestion. Validate its category, handle timeouts, and send low-confidence or invalid results to a person. Avoid storing private ticket text unless there is a clear need.

The sequence is simple: define the user problem, build the smallest useful baseline, measure it, then let a model prove it improves the result.

Stack & Craft shares practical engineering guides at https://stackandcraft.in/ .