Taking manual reply triage off users' hands with an LLM classifier

How I put intent classification into production on an AI sales-automation platform, as its sole engineer, and why the manual path stayed.

The manual process

The platform is an AI sales-automation product: users run outreach campaigns, and prospects reply. Those replies are where the value is, and before the classifier every one of them was sorted by hand. Users went through a campaign's replies in the dashboard and decided for each one what it was: is this person interested, not interested, away, or agreeing to a meeting? At peak the platform handled close to 100,000 requests a day, busiest in the evening, and roughly 5,000 to 10,000 of them were replies to sort.

When I joined, I worked on that manual categorization in the messages view. Then I was the one who replaced it.

What the category decides

Sorting replies wasn't bookkeeping. The category decides what the platform does next:

• Interested: an AI-generated reply goes out.
• Not interested: the contact goes on a list the next campaign leaves out.
• Out of office: an email is scheduled for the date the person is back.
• Meeting scheduled: the reply agreed to a meeting.
• A catch-all category for anything that fits none of them.

That's 5 of the 8 categories I still remember. When I took over, these actions existed but were completely broken, and the conditions behind them were buried in code that was hard to follow. I rebuilt them into working order, then put the classifier in front of them.

The point is that a wrong label is never just a wrong label: it sends the wrong next action to a real person. A not-interested reply filed as out-of-office gets a follow-up email from someone who already said no.

How the classifier works

I built it on Azure OpenAI, with four decisions doing most of the work.

A fixed label list with structured output. The model doesn't write a description of the reply; it returns one of the known categories in a fixed structure. Free text would have needed parsing, and parsing is where silent failures hide.

Examples in the prompt. The prompt carries example replies for the categories, so the model sees what "interested" looks like in a real sales inbox, not a dictionary definition of the word.

A catch-all instead of a forced guess. Anything outside the list falls to the catch-all. A reply the model can't place lands where a person will look at it, instead of being forced into the nearest category and triggering the wrong action.

One wrapper around the provider. Every OpenAI API call in the product goes through one wrapper pointed at an Azure OpenAI deployment, so changing provider or deployment is a config edit, not a change at every call site.

Keeping the human path

The manual categorization didn't go away. It became the correction path: when the classifier gets one wrong, the user moves the reply to the right category, exactly as they used to. The decision was made with the client, and it was the right one: users went from sorting every reply to fixing the few the model got wrong.

The mistakes I remember seeing most were not-interested replies tagged out-of-office, and replies the model put in the catch-all. The first is the costly kind: it sends a follow-up to someone who already said no.

Running it in production

The classifier sat inside a larger system I owned end to end for the whole tenure: the React dashboard, the Node.js and GraphQL backend on MongoDB, and the BullMQ and Redis queue that runs outreach. That queue is where production showed its teeth: a 5-minute stall backed up thousands of jobs, and all of them drained afterward with none dropped. An LLM feature is only as reliable as the pipeline it runs in.

What I'd add today

Looking back with what I know now, the correction path was the most valuable part of the design, and it was underused. Every time a user moved a reply to a different category, they produced a labelled example of exactly the kind of reply the model gets wrong. Today I'd log every correction and use them two ways: as an evaluation set, run against every prompt or model change before it ships, and as fresh examples for the prompt. The classifier would then get better from the same clicks users were already making.

The pattern

Strip away the specifics and this is the shape of most useful LLM work in a business:

1. Find the decision people make by hand, and what each decision triggers.
2. Constrain the model to those decisions: fixed labels, structured output, examples from real data.
3. Give it a way to say "I don't know" that lands in front of a person.
4. Keep the manual path as the correction path, and treat each correction as data.

That's the work I do: taking a manual process and turning it into an AI feature that runs in production and keeps working once people depend on it.

The platform: b2brocket.ai

Awais Shahid, Senior Full-Stack AI Engineer. LinkedIn · GitHub