Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 8 min read

Your AI found a customer 'trend'. Check it in Python before it reaches a slide

This article was created with the help of AI. I ran and tested the code before publishing (Python 3.13.5, standard library only, 30/30 tests passing) and cross-checked the interval maths against statsmodels 0.15.0. All t

This article was created with the help of AI. I ran and tested the code before publishing (Python 3.13.5, standard library only, 30/30 tests passing) and cross-checked the interval maths against statsmodels 0.15.0. All the data in it is invented.

LLMs are now routinely used to tag customer calls, chats and tickets with themes ("invoice confusion", "login failure", "wants feature X"). The output usually ends up as a ranked list or a chart, and from there into a product or marketing decision.

The problem is that a ranked list hides everything you need to know to trust it:

  • Out of how many? A count without a denominator, and without saying how many conversations were actually analysed.
  • Did anyone say that? LLMs can produce fluent text that isn't grounded in the input. NIST's Generative AI Profile calls this "confabulation" (NIST AI 600-1, section 2.2). In theme tagging it shows up as a supporting "quote" that isn't in the transcript.
  • Is the tag right? Pangakis, Wolken and Fasching (2023) replicated 27 annotation tasks with GPT-4 and found performance "highly contingent on both the dataset and the type of annotation task"; their recommendation is to validate LLM labels against human labels, task by task.
  • Is it noise? Six mentions out of 110 has a wide uncertainty range, and "up from zero" is easy to put on a slide.
  • Is it one person? Four of those six might be the same customer.

This post builds a small gate that answers those questions for every theme and gives each a verdict: READY, CAVEAT or NOT_READY. It doesn't call an LLM. It sits after whatever tagging step you already have.

Input format

One JSON file. Each conversation has an id, a period, a customer id, a transcript, and the AI's tags, each with the quote the AI says supports it. An optional audit list holds human checks of individual tags.

{
  "conversations": [
    {"id": "2026-11-0105", "period": "2026-11", "customer_id": "cust-0042",
     "analysed": true,
     "transcript": "Agent: ... Customer: can you add a Spanish option to the phone menu.",
     "themes": [{"theme": "wants Spanish menu",
                  "quote": "can you add a Spanish option to the phone menu"}]},
    {"id": "2026-11-0110", "period": "2026-11", "customer_id": "cust-8812",
     "analysed": false}
  ],
  "audit": [
    {"conversation_id": "2026-11-0105", "theme": "wants Spanish menu", "human_agrees": true}
  ]
}

analysed: false conversations count toward the period total but not the denominator, so coverage is visible instead of silently dropped.

Step 1: verify quotes against the transcript

Exact substring matching is too brittle (curly apostrophes, double spaces, punctuation), but anything fuzzier starts accepting paraphrases. A middle ground: normalise both sides (Unicode NFKC, casefold, strip punctuation, squash whitespace) and then require a substring match.

_PUNCT = re.compile(r"[^\w\s]", re.UNICODE)
_WS = re.compile(r"\s+")


def normalise(text: str) -> str:
    """NFKC-normalise, casefold, drop punctuation, squash whitespace."""
    text = unicodedata.normalize("NFKC", text).casefold()
    text = _PUNCT.sub(" ", text)
    return _WS.sub(" ", text).strip()


def quote_found(quote: str, transcript: str) -> bool:
    q = normalise(quote)
    return bool(q) and q in normalise(transcript)

Tags whose quote isn't found are excluded from the count and reported. If more than 5% of a theme's quotes fail, the theme is NOT_READY: that's a signal about the tagging prompt or pipeline, not just a few bad rows.

Step 2: a share with an honest range

For "k of n conversations", use the Wilson score interval rather than the textbook p ยฑ zยทโˆš(p(1โˆ’p)/n), which can go below zero and behaves badly for small counts. The NIST/SEMATECH e-Handbook (7.2.4.1) gives the formula and notes the lower limit can't be negative.

def wilson(successes: int, n: int, z: float = Z95) -> tuple[float, float]:
    """Wilson score interval for a binomial proportion (NIST e-Handbook 7.2.4.1)."""
    if n < 0 or successes < 0 or successes > n:
        raise ValueError("need 0 <= successes <= n")
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z * z / n
    centre = p + z * z / (2 * n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, (centre - half) / denom), min(1.0, (centre + half) / denom))

The same function gives a range for audited precision: if a person checked 25 tags and agreed with 24, precision is 96% with a 95% interval of about 80% to 99%.

Step 3: is it actually changing?

Comparing two months by eyeballing two intervals is unreliable (overlapping intervals don't mean "no difference"). Instead, build an interval for the difference in proportions. Newcombe's hybrid score method (Newcombe 1998) combines the two Wilson intervals:

def newcombe_diff(x1: int, n1: int, x2: int, n2: int, z: float = Z95) -> tuple[float, float, float]:
    """Difference p1 - p2 with Newcombe's hybrid score interval (Newcombe 1998)."""
    if n1 == 0 or n2 == 0:
        return (float("nan"), -1.0, 1.0)
    p1, p2 = x1 / n1, x2 / n2
    l1, u1 = wilson(x1, n1, z)
    l2, u2 = wilson(x2, n2, z)
    d = p1 - p2
    lower = d - math.sqrt((p1 - l1) ** 2 + (u2 - p2) ** 2)
    upper = d + math.sqrt((u1 - p1) ** 2 + (p2 - l2) ** 2)
    return (d, max(-1.0, lower), min(1.0, upper))

If the interval is entirely above zero, the theme is rising; entirely below, falling; otherwise no clear change.

Step 4: the verdict

The rules are deliberately boring and configurable. Hard failures make a theme NOT_READY; soft ones make it CAVEAT.

def judge(tp: ThemePeriod, cfg: dict) -> None:
    hard, soft = [], []
    if tp.tags and tp.unverified_quotes / tp.tags > cfg["max_unverified_rate"]:
        hard.append(f"{tp.unverified_quotes}/{tp.tags} supporting quotes not found in transcripts")
    if tp.verified_mentions < cfg["min_mentions"]:
        hard.append(f"only {tp.verified_mentions} verified mentions (min {cfg['min_mentions']})")
    if tp.precision_ci is not None and tp.audited >= cfg["min_audited"] \
            and tp.precision_ci[0] < cfg["min_precision_lower"]:
        hard.append(f"audited precision lower bound {tp.precision_ci[0]:.2f} < {cfg['min_precision_lower']}")
    if tp.audited < cfg["min_audited"]:
        soft.append(f"only {tp.audited} tags human-audited (min {cfg['min_audited']})")
    if tp.coverage < cfg["min_coverage"]:
        soft.append(f"only {tp.coverage:.0%} of conversations analysed")
    if tp.verified_mentions and tp.top_customer_share > cfg["max_top_customer_share"]:
        soft.append(f"one customer accounts for {tp.top_customer_share:.0%} of mentions")
    tp.reasons = hard + soft
    tp.verdict = "NOT_READY" if hard else "CAVEAT" if soft else "READY"

Defaults: at least 10 verified mentions, at least 20 human-audited tags, a precision lower bound of 0.70, at least 80% of conversations analysed, no single customer behind more than 30% of mentions, and no more than 5% unverifiable quotes. These are starting points, not standards. Tune them to how expensive the decision is.

Note what is not a hard failure: a small audit. Too few audited tags means "we don't know yet", which is a caveat. A precision lower bound below 0.70 on an adequate audit means "we know it's unreliable", which blocks.

Running it on the sample

sample/conversations.json is generated by make_sample.py (fixed seed, invented data): 104 conversations in October (100 analysed), 120 in November (110 analysed), four themes.

$ python voc_evidence.py sample/conversations.json --period 2026-11 --baseline 2026-10
[CAVEAT] delivery delay (2026-11)
  15 of 110 analysed conversations (13.6%, 95% CI 8.4%-21.3%); 110/120 analysed; 15 customers
  human audit: 7/8 confirmed (precision 88%, 95% CI 53%-98%)
  vs 2026-10: -0.4% (95% CI -10.0% to +9.0%) -> no clear change
  - only 8 tags human-audited (min 20)
[READY] invoice confusion (2026-11)
  26 of 110 analysed conversations (23.6%, 95% CI 16.7%-32.4%); 110/120 analysed; 26 customers
  human audit: 24/25 confirmed (precision 96%, 95% CI 80%-99%)
  vs 2026-10: +11.6% (95% CI +1.2% to +21.7%) -> rising
[NOT_READY] login failure (2026-11)
  10 of 110 analysed conversations (9.1%, 95% CI 5.0%-15.9%); 110/120 analysed; 10 customers
  human audit: 10/14 confirmed (precision 71%, 95% CI 45%-88%)
  vs 2026-10: +0.1% (95% CI -8.2% to +8.1%) -> no clear change
  - 4/14 supporting quotes not found in transcripts
  - only 14 tags human-audited (min 20)
[NOT_READY] wants Spanish menu (2026-11)
  6 of 110 analysed conversations (5.5%, 95% CI 2.5%-11.4%); 110/120 analysed; 3 customers
  human audit: 6/6 confirmed (precision 100%, 95% CI 61%-100%)
  vs 2026-10: +5.5% (95% CI +0.7% to +11.4%) -> rising
  - only 6 verified mentions (min 10)
  - only 6 tags human-audited (min 20)
  - one customer accounts for 67% of mentions
exit=1

What each case demonstrates:

  • invoice confusion passes everything and is clearly rising.
  • delivery delay looks fine, but only 8 tags were audited, so it carries a caveat.
  • login failure: 4 of its 14 tagged conversations cite a "quote" that isn't in the transcript. They're dropped from the count, and the theme is blocked until someone looks at the pipeline.
  • wants Spanish menu is statistically "rising" (from zero), and that is exactly why the other checks exist: 6 mentions, 3 customers, one of them behind 67%.

Exit code is 1 when any theme is NOT_READY (2 for input errors), so you can run it in CI or a scheduled job before a report is published. --json gives machine-readable output.

Tests

test_voc_evidence.py (unittest, 30 tests) covers: Wilson and Newcombe values against reference numbers from statsmodels 0.15.0 (proportion_confint(method="wilson"), confint_proportions_2indep(method="newcomb")), interval edge cases, quote normalisation (case, punctuation, curly apostrophes, paraphrase rejection), denominator and coverage, duplicate tags in one conversation counted once, each verdict rule and its threshold boundary, period comparison, input validation and CLI exit codes.

$ python -m unittest test_voc_evidence
Ran 30 tests in 0.007s

OK

Limitations (read before using)

  • Quote matching is lexical. It proves the words appear; it doesn't prove the tag is right (that's what the audit is for), or that the words came from the customer rather than the agent. If your transcripts have speaker labels, match within customer turns only.
  • Transcripts can be wrong too. For calls, you are verifying against speech-to-text output, not audio.
  • Precision only. The audit measures whether tags are right, not what the tagger missed. Sample untagged conversations occasionally to estimate recall.
  • Statistical assumptions. Intervals assume conversations are independent. Repeat contacts from the same customer violate that, which is one reason concentration is reported separately. Checking many themes every month will also produce some false "rising" flags by chance.
  • Selection. Everything here describes people who contacted you, in the channels you analysed. It says nothing about customers who never got in touch.
  • Thresholds are judgment calls, not industry standards.
  • Privacy. Transcripts are personal information. Run this where your existing access controls apply, and check that analysing conversations for product insight fits the purposes you told customers about.

Files: voc_evidence.py, test_voc_evidence.py, make_sample.py, sample/conversations.json.

Sources: Pangakis, Wolken and Fasching, Automated Annotation with Generative AI Requires Validation (2023); NIST, AI 600-1 Generative AI Profile (2024); NIST/SEMATECH, e-Handbook of Statistical Methods 7.2.4.1; Newcombe, Interval estimation for the difference between independent proportions, Statistics in Medicine 17(8), 1998.

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.