Cost-Aware Abstention: Why Autonomous Ticket Routing Demands Calibrated Rejection Over Raw LLMs

In enterprise IT service desks and CRM architectures, automated routing is frequently touted as an easy win for Large Language Models. Teams wire up zero-shot prompts to incoming ticket streams, connect webhook dispatches, and celebrate instantaneous automation.

However, in production workflows, routing errors exhibit steep cost asymmetry. When a ticket is dispatched to the wrong engineering team, it languishes across multiple handoffs, inflates mean time to resolution (MTTR), breaches enterprise SLAs, and requires human triage to untangle. Recent empirical research (arXiv:2610.31657) proves that deploying LLMs without cost-aware abstention policies is an architectural failure mode that directly degrades unit economics.

1. The Cost Asymmetry Problem

Consider an explicit operational cost model defined by a triplet $(c_{ok}, c_{triage}, c_{wrong})$:

• $c_{ok} = 0$: Correct automated ticket routing incur minimal marginal cost.
• $c_{triage} = 1$: Deferring the ticket to a human dispatcher costs baseline manual handling.
• $c_{wrong} = 5$: Misrouting the ticket incurs operational churn, cross-team friction, and re-triage.

Under this cost surface, routing a ticket only makes economic sense if the probability of correctness $p$ satisfies the break-even condition:

$$p \cdot c_{ok} + (1 - p) \cdot c_{wrong} \le c_{triage}$$

Substituting our baseline values:

$$(1 - p) \cdot 5 \le 1 \implies p \ge 1 - \frac{1}{5} = 0.80$$

If the system's calibrated confidence is below $0.80$, attempting an automated decision is mathematically guaranteed to generate negative expected value compared to human triage.

2. Empirical Findings: The LLM Calibration Blindspot

Benchmarking four non-LLM classifiers against four industry-standard LLMs (including GPT-4o-mini and Claude Haiku) across 47,837 tickets revealed severe systemic deficiencies in prompt-based dispatchers:

1. Uncalibrated Overconfidence: Zero- and few-shot LLM endpoints exhibited correctness AUROC around $0.500$—equivalent to a random guess when predicting whether their own output was correct. Isotonic recalibration failed because endpoints produced near-constant confidence clusters.
2. All-or-Nothing Deferral: When subjected to the analytically derived break-even threshold ($p^* = 0.80$), seven prompted LLM configurations deferred 100% of tickets, while Haiku few-shot routed only 19 out of 1,000.
3. Small Dedicated Classifiers Outperform: A fine-tuned DistilBERT reached $0.879$ accuracy on held-out evaluation data, compared to only $0.477$ for the best-performing prompted LLM. A basic TF-IDF with Logistic Regression, tuned to a frozen decision boundary of $0.60$ on calibration data, slashed test costs by $38.1\%$ and reduced misrouted tickets from $15.3\%$ to $3.5\%$.

3. Architecture Pattern: The Selective Abstention Router

Production multi-agent CRM systems must decouple triage classification from autonomous routing execution via a deterministic, cost-aware guardrail:

from dataclasses import dataclass
from typing import Optional, Protocol
import numpy as np

@dataclass(frozen=True)
class RoutingDecision:
    assigned_queue: Optional[str]
    confidence: float
    abstained: bool
    reason: str

class CalibratedClassifier(Protocol):
    def predict_proba(self, text: str) -> np.ndarray: ...
    @property
    def classes_(self) -> list[str]: ...

class CostAwareTicketRouter:
    def __init__(
        self,
        classifier: CalibratedClassifier,
        c_ok: float = 0.0,
        c_triage: float = 1.0,
        c_wrong: float = 5.0
    ):
        self.classifier = classifier
        self.classes = classifier.classes_
        # Analytical break-even threshold
        self.break_even_threshold = (c_wrong - c_triage) / (c_wrong - c_ok)

    def route(self, ticket_body: str) -> RoutingDecision:
        probabilities = self.classifier.predict_proba(ticket_body)
        max_idx = int(np.argmax(probabilities))
        max_prob = float(probabilities[max_idx])
        predicted_queue = self.classes[max_idx]

        if max_prob < self.break_even_threshold:
            return RoutingDecision(
                assigned_queue=None,
                confidence=max_prob,
                abstained=True,
                reason=f"Confidence {max_prob:.3f} below break-even {self.break_even_threshold:.2f}"
            )

        return RoutingDecision(
            assigned_queue=predicted_queue,
            confidence=max_prob,
            abstained=False,
            reason="Confidence cleared break-even barrier"
        )

4. Production Engineering Takeaways

• Never let an LLM act as a solitary, uncalibrated router when misrouting penalty $c_{wrong} \gg c_{triage}$.
• Freeze calibrated thresholds offline: Calibrate rejection thresholds on historical holdout partitions rather than dynamically guessing confidence in runtime prompts.
• Deploy hybrid cascades: Use lightweight, fine-tuned transformer or linear classifiers as calibrated dispatchers. Reserve expensive LLMs for context enrichment or synthetic drafting, not high-stakes discrete routing.

Originally published at https://bohdanhordii.dev

Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan_hordii