Log Every "I Don't Know": A Tiny Python Webhook That Shows When a Packaged Chatbot Has Hit Its Limits

Most packaged chatbot tools can send a webhook when a conversation event happens. If yours can fire one on a fallback (the bot did not understand) or a handover request, you can collect hard evidence for a decision many owners make on gut feel: whether to go with custom AI or stay with an off-the-shelf chatbot.

Here is a tiny receiver that appends those events to a CSV.

The receiver

import csv, json
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, HTTPServer

LOG = "fallbacks.csv"
WANTED = {"fallback", "handover_requested"}

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        try:
            event = json.loads(self.rfile.read(length) or b"{}")
        except json.JSONDecodeError:
            self.send_response(400); self.end_headers(); return
        if event.get("type") in WANTED:
            with open(LOG, "a", newline="") as f:
                csv.writer(f).writerow([
                    datetime.now(timezone.utc).isoformat(),
                    event.get("type"),
                    event.get("channel", ""),
                    (event.get("last_user_message") or "")[:300],
                ])
        self.send_response(204); self.end_headers()

HTTPServer(("0.0.0.0", 8080), Hook).serve_forever()

Map the field names to whatever your tool sends. Put it behind HTTPS and a shared secret before using it with real traffic, and do not store more of the message than you need.

Reading the log

After a few weeks, group the last_user_message column by hand into a few buckets:

• Missing answer. "Do you deliver on Sundays?" Add it to the bot's knowledge. No build needed.
• Needs live data. "Is my order out for delivery?", "Any slots tomorrow?" The bot needs to read your systems.
• Needs judgement. Complaints, special requests. These should go to a person quickly.

If the second bucket keeps growing, the packaged tool is reaching its limits. That is the point where a custom build connected to your own systems is worth discussing.

I work on these builds at NxFlowAI. Before any of that, we do a 72-hour audit of one workflow; the step-by-step workflow audit guide on our blog shows the same steps you can do yourself.