Building a WhatsApp AI Chatbot That Hands Off to a Human: Architecture and Pitfalls

Getting a language model to answer WhatsApp messages takes an afternoon. Getting it to stop answering at the right moment, pass the conversation to a person with enough context, and never talk over that person afterwards takes considerably longer. Most of the bugs we have seen in production chatbots are not about answer quality. They are about state: who owns the conversation right now, which messages have already been processed, and what happens when the phone, the model or the manager goes quiet.

This post walks through an architecture that holds up under real traffic, with the parts that usually break.

The shape of the system

At a high level there are five components:

1. Transport. Receives incoming messages and sends replies. This is either the official WhatsApp Business Platform with webhooks, or a session linked to an existing number through Linked Devices (the same mechanism WhatsApp on a computer uses).
2. Inbox queue. Every incoming event goes into a durable queue before anything else touches it.
3. Conversation store. One record per chat: owner (bot or human), recent history, flags, timestamps.
4. Assistant worker. Pulls from the queue, builds the prompt from the knowledge base and the history, decides between "answer" and "hand off".
5. Operator side. Wherever managers actually reply: the phone itself, a shared inbox, or a CRM.

The rule that saves you later: the transport layer does nothing except normalise events and enqueue them. No model calls inside a webhook handler. Webhook senders retry on slow responses, and an LLM call that takes eight seconds will happily produce duplicate replies.

Normalise and deduplicate first

Whatever transport you use, convert events into one internal shape:

from dataclasses import dataclass

@dataclass
class InboundEvent:
    message_id: str      # provider's ID, used for dedup
    chat_id: str         # stable per customer
    from_me: bool        # sent from the business number
    kind: str            # "text", "voice", "image", "document"
    text: str | None
    media_ref: str | None
    ts: int              # provider timestamp, seconds

Then deduplicate on message_id before enqueueing. A Redis SET NX with a TTL of a day is enough:

def accept(event: InboundEvent) -> bool:
    key = f"seen:{event.message_id}"
    if not redis.set(key, 1, nx=True, ex=86400):
        return False          # duplicate delivery, drop it
    queue.push(event)
    return True

Webhook retries, reconnects of a linked session and your own worker restarts will all deliver the same message twice. Without dedup the customer gets two slightly different answers to one question.

Note the from_me field. On a linked number, messages the manager types on the phone arrive as events too. You need them, because they are the most reliable signal that a human has taken over (more on that below).

Debounce bursts before calling the model

People do not write in single messages. They write "hi", then "how much is", then "delivery to Almaty?" within ten seconds. If the worker answers each fragment, you get three replies, the first two of them useless.

A simple debounce: when a message arrives for a chat, schedule processing a few seconds later and reset the timer on every new message.

DEBOUNCE_SEC = 4

def on_event(event):
    store.append_pending(event.chat_id, event)
    scheduler.schedule(
        job_id=f"reply:{event.chat_id}",
        run_at=now() + DEBOUNCE_SEC,
        replace_existing=True,
    )

When the job fires, it takes all pending messages for that chat as one turn. Three to five seconds is the usual sweet spot: shorter and you still split sentences, longer and the bot feels sluggish. Voice notes need special handling here, because transcription adds its own delay; start the debounce timer from the moment the transcript is ready, not from the moment the audio arrived.

Voice notes are text with a confidence problem

A large share of WhatsApp traffic is voice. The pipeline is straightforward: download the media, transcribe it, treat the transcript as a text message. The pitfall is noise. A voice note recorded on a busy street produces a transcript that is grammatically fine and semantically wrong, and the model will confidently answer the wrong question.

Two mitigations that work:

• Keep the per-segment confidence or average log-probability from the speech model and, below a threshold, have the assistant ask the customer to clarify instead of answering.
• Store the original audio reference with the transcript, so a manager who takes over can listen to it rather than trust the text.

Conversation ownership is a state machine

This is the core of the handoff design. Each chat has exactly one owner at a time:

            customer asks for human / trigger fires
   BOT  ───────────────────────────────────────────▶  PENDING_HUMAN
    ▲                                                     │
    │                                     manager replies │
    │                                                     ▼
    └──────────── idle timeout or manual release ─────  HUMAN

• BOT: the assistant answers.
• PENDING_HUMAN: the assistant has told the customer a person will reply, and stays silent. Managers are notified.
• HUMAN: a manager is active. The assistant is fully muted for this chat.

Store it explicitly and check it at the last possible moment, right before sending, not only when the job starts:

def send_reply(chat_id, text, generated_at_version):
    conv = store.get(chat_id)
    if conv.owner != "BOT":
        return  # a human took over while the model was thinking
    if conv.version != generated_at_version:
        return  # new messages arrived, this answer is stale
    transport.send(chat_id, text)

The version check matters more than it looks. A model call can take several seconds; during that time the customer may add a message or a manager may jump in. Sending a reply generated from an outdated state is the single most common source of "the bot interrupted me" complaints from managers.

What triggers a handoff

Hard-coding "if the message contains 'operator'" catches only a fraction of cases. A better approach is to let the model return a structured decision alongside the answer:

{
  "action": "handoff",
  "reason": "custom_terms",
  "summary": "Wants 40 units with installation in Shymkent, asks for a discount on volume.",
  "reply_to_customer": "Thanks, I'm passing this to a manager who can confirm volume terms."
}

Typical handoff reasons worth defining up front:

• the customer explicitly asks for a person;
• the customer is ready to buy and the next step needs a human (payment, contract, scheduling);
• non-standard terms: discounts outside the price list, unusual quantities, custom deadlines;
• complaints and negative sentiment;
• the question is outside the knowledge base after one clarification attempt.

Make the list configurable per business rather than baked into the prompt. A clinic and a furniture shop draw the line in very different places.

Equally important are hard boundaries that never depend on the model's judgement. The assistant should not promise discounts beyond the price list, quote deadlines that depend on workload, or give professional conclusions. Encode these both in the system prompt and as post-generation checks: a regex for percentages and currency amounts that do not appear in the knowledge base catches a surprising number of invented prices.

Detecting that a human has taken over

If managers reply from a CRM or a shared inbox, you know when they send something. If they reply from the phone itself, which is common with small teams, you only know through from_me events from the transport.

The rule: any from_me message that was not sent by the bot moves the chat to HUMAN. To tell the bot's own messages apart, record the provider message IDs of everything the bot sends:

def on_from_me(event):
    if redis.sismember(f"bot_sent:{event.chat_id}", event.message_id):
        return                      # our own echo
    store.set_owner(event.chat_id, "HUMAN", since=event.ts)

Do not rely on comparing message text; managers copy bot phrases more often than you would expect.

Giving the manager context

A handoff that dumps the customer on a person with no context forces them to ask everything again, which defeats the purpose. Send the manager:

• the model's one-paragraph summary from the handoff decision;
• the reason code;
• the last N messages verbatim, including transcripts of voice notes;
• anything already collected: name, city, items and quantities mentioned.

The summary is generated once at handoff time and stored, so it stays consistent if several managers look at the chat.

Releasing the chat back to the bot

The forgotten transition is HUMAN back to BOT. If you never release, the bot slowly goes silent across your whole customer base. If you release too eagerly, the bot answers in the middle of a human negotiation.

What works in practice:

• An idle timeout measured from the last manager message, not the last customer message. Something between 2 and 12 hours depending on the business.
• A manual release command from the operator side.
• On release, the bot does not proactively write anything. It simply answers the next incoming message, with the human part of the conversation included in its history so it does not contradict what the manager promised.

That last point is easy to miss. If the manager agreed to a delivery date, the bot needs to see that agreement, or it will cheerfully quote the standard terms on the next question.

Knowledge base instead of a decision tree

The reason menu-driven bots age badly is maintenance: every new service or price change means editing the tree. With an LLM, the price list and service descriptions become the knowledge base, and a new line in the price list is available immediately.

For a typical small business catalogue (a few hundred items) you often do not need vector search at all. The whole price list fits in the context window, and putting it there directly removes a class of retrieval failures where the right item simply is not fetched. Move to retrieval when the catalogue grows past what fits comfortably, and even then keep exact-match lookup for SKUs and product names alongside embeddings.

Feed corrections back from real conversations. Log every handoff with reason outside_knowledge_base; reviewing that list weekly is the cheapest way to improve answers.

Operational pitfalls

The linked phone goes offline. With Linked Devices the link runs through the phone. If it loses connectivity, the session stops receiving messages. Monitor last-event time per number and alert when it exceeds a sensible window during working hours. When the link comes back, the replayed backlog hits your dedup layer, which is another reason that layer has to exist.

Bulk messaging gets numbers banned. Answering inbound messages on a business number is normal usage. Sending to cold lists is what triggers restrictions. Keep the bot strictly reactive.

Late answers. If processing fails and recovers an hour later, do not answer an hour-old question as if nothing happened. Drop or rephrase replies to messages older than a cutoff, and route those chats to a human instead.

Several numbers. Branches often have different prices. Key the conversation store and the knowledge base by number, and allow a shared base with per-number overrides.

Logging and privacy. Decide retention up front and mask phone numbers in application logs.

A checklist before going live

• Dedup on provider message ID before the queue
• Debounce of 3 to 5 seconds per chat, started after transcription for voice
• Explicit owner state per chat, checked again right before sending
• Version check to drop stale model answers
• Structured handoff decision with reason and summary
• from_me detection with a record of bot-sent IDs
• Idle-based release measured from the last manager message
• Hard checks against invented prices and deadlines
• Alert on silent numbers, cutoff for late answers
• A sandbox chat where answers are reviewed before the first real customer

None of these pieces is complicated on its own. The difficulty is that skipping any one of them produces failures that look like "the AI said something weird", when the real cause is a race between a model call and a manager's thumb. If you would rather start with a setup that already handles connection, voice notes and handoff, and test it on your own price list first, take a look at our WhatsApp AI chatbot that works on your existing number.