A customer asked about the warranty four times in one chat. Your AI gave four different answers.

A regional appliance repair company — 6 vans, one shared support inbox, one AI answering first — settled a small-claims case last spring over a $740 compressor replacement. Not because the repair was bad. Because of what their own chat transcript said.

The customer's thread was 22 messages long and spanned nine days. Four times, in four different phrasings, she asked the same question: how long is the warranty on this repair?

• Message 4: "All repairs carry a 12-month parts and labor warranty."
• Message 9 (asked again after a scheduling delay): "Your repair is covered for 24 months."
• Message 15 (asked about the compressor specifically): "The compressor itself has a lifetime warranty."
• Message 21 (asked one more time, plainly): "12 months from the date of service."

She filed the claim in month 14, quoting message 9 and message 15. The company's defense was that the policy is 12 months. Her reply, which the judge accepted, was that the company's own agent had said otherwise twice, in writing, and she had relied on it.

The owner's read was "the AI made stuff up." That's not what happened. The AI answered correctly every single time — correctly for the chunk it retrieved at that moment. Twelve months lived in warranty_policy.md. Twenty-four months lived in a legacy promo page from 2023 that was still indexed. "Lifetime on the compressor" was a line from a manufacturer's parts sheet that had been ingested as if it were the company's own policy.

Four different documents, four different retrieval wins, four confident answers, one thread. The model didn't drift because it was broken. It drifted because nothing in the system remembered what it had already told this customer.

In-thread answer drift is a different bug from being wrong

Most teams test their AI by asking one question and grading one answer. That test cannot find this failure, because every individual answer passes.

The failure only appears on the second asking. And customers ask twice constantly — when they're anxious, when a date slipped, when they're building a case, when they simply didn't believe the first answer. At this company, across 1,860 conversations in one quarter, 214 threads (11.5%) contained the same question asked three or more times. In 71 of those — 33% — the AI produced at least two different answers. In 19 threads it produced three or more. Every one of those 19 is a potential small-claims exhibit.

Three mechanisms produce it, and none of them is "hallucination":

1. Retrieval variance. A rephrased question embeds differently, so it matches a different chunk. "How long is the warranty" and "what about the compressor specifically" are not near-duplicates to a vector index — they're different queries with different best matches.
2. No conversational memory of commitments. The model sees the prior turns, but nothing in its instructions says those turns are binding. It treats its own earlier answer as context, not as a record.
3. Reward for freshness. Ask a model the same thing twice and it tends to produce a new answer — a rephrase, an elaboration, an added exception. Novelty reads as helpfulness. In a warranty conversation, novelty is liability.

There's a fourth, quieter mechanism: the fix for one answer creates the next drift. Someone notices "24 months" was wrong and edits the promo page. Now the same question returns 12 months, and the customer who was told 24 has a documented contradiction with a clean audit trail pointing at your CMS.

Step 1: An answer ledger, not a memory

The fix is not "make the model more consistent." Temperature 0 does not solve retrieval variance, and it does not create an obligation. The fix is to treat every answer that creates a commitment as a record, keyed by the question, written at first answer, and replayed on every repeat.

{
  "answer_ledger_entry": {
    "entry_id": "ans_2026_03_11_0094",
    "conversation_id": "conv_77213",
    "customer_id": "cust_40912",
    "question_key": "warranty_duration::repair_general",
    "question_canonical": "How long is the warranty on a completed repair?",
    "matched_by": "semantic_cluster",
    "answer_text": "All repairs carry a 12-month parts and labor warranty from the service date.",
    "answer_source": {
      "doc_id": "warranty_policy_v4",
      "section": "2.1 Coverage period",
      "effective_from": "2025-01-01",
      "retrieved_score": 0.91
    },
    "commitment_class": "binding_policy",
    "locked": true,
    "locked_at": "2026-03-11T09:14:22Z",
    "supersedes": null,
    "replay_count": 3,
    "correction_event": null
  },
  "ledger_rules": {
    "same_question_key_returns_locked_answer_verbatim": true,
    "retrieval_may_not_override_a_locked_answer": true,
    "answer_source_must_resolve_to_one_document_version": true,
    "conflicting_documents_must_be_resolved_before_first_answer": true,
    "correction_requires_a_stated_reason_and_a_human_owner": true
  }
}

Four fields do the work.

question_key is the lock's address. Not the raw text — a canonical key from a small, curated list of commitment topics: warranty duration, cancellation window, refund eligibility, response SLA, price validity, deposit terms. You do not need an ontology. You need about twenty keys, because commitments cluster tightly around twenty questions.

locked: true is the whole idea. Once answered, the answer for this customer and this topic is frozen for the life of the thread. Repeat questions get the locked text back — not a re-retrieval, not a rephrase, not an elaboration. If the customer asks "but what about the compressor," the agent may add scope detail from the same source document, but it may not introduce a second document's duration.

answer_source with effective_from is what kills the legacy-page problem. An answer is only allowed if exactly one active document version backs it. If two active documents disagree on the same key, retrieval doesn't get to pick a winner — the gate fires before the first answer is ever written.

replay_count turns consistency into something you can measure. A healthy system has threads with replay_count up to 5 and exactly one distinct answer_text. That's the metric, and it's trivially queryable.

Step 2: The gate

{
  "consistency_gate": {
    "trigger": "inbound_message_resolving_to_a_commitment_question_key",
    "branches": {
      "first_ask_key_resolved_single_active_source": "retrieve_answer_write_ledger_lock_answer",
      "first_ask_multiple_active_sources_conflict": "suppress_answer_route_to_policy_owner_same_day",
      "first_ask_no_source_found": "state_unknown_offer_human_callback_never_improvise",
      "repeat_ask_ledger_entry_exists": "replay_locked_answer_verbatim_no_new_retrieval",
      "repeat_ask_paraphrased_to_adjacent_key": "replay_parent_key_then_name_the_boundary",
      "repeat_ask_customer_asserts_a_different_answer": "quote_ledger_verbatim_offer_written_copy_do_not_argue",
      "repeat_ask_with_new_facts_changing_applicability": "route_to_human_correction_protocol",
      "policy_changed_mid_thread": "honor_locked_answer_for_this_customer_apply_new_policy_going_forward"
    },
    "forbidden": [
      "answering_a_repeat_question_from_fresh_retrieval",
      "introducing_a_second_source_for_a_key_already_locked",
      "silently_replacing_an_earlier_answer_without_a_recorded_correction",
      "letting_a_second_document_win_a_tie_at_answer_time",
      "rephrasing_a_locked_answer_into_a_different_commitment"
    ]
  }
}

Three branches earn their keep.

repeat_ask_paraphrased_to_adjacent_key is the actual bug at this company. "What about the compressor specifically" is an adjacent key, and the naive system treats it as a fresh question — which is how a manufacturer's parts sheet became a company warranty promise. Naming the boundary ("the compressor is included in the 12-month coverage; there is no separate lifetime term") closes the hole without a second commitment.

repeat_ask_customer_asserts_a_different_answer happens more than teams expect, and it splits two ways. Sometimes the customer is misremembering and the ledger is clean — then quote the entry verbatim and offer the transcript, without arguing. Sometimes the customer is right and the ledger shows your own drift — then you need the correction protocol, not a debate.

policy_changed_mid_thread is the one that looks unfair and isn't. If the warranty went from 12 to 18 months on April 1, a customer whose repair was quoted under the 12-month policy in March keeps 12 months. The gate honors the lock and applies the new policy only to future threads. That is both the legally safer and the operationally simpler rule.

Step 3: A correction protocol, because sometimes you must change an answer

Locking answers without a correction path just trains the system to defend a wrong answer forever. That's worse. The correction must be possible — and it must be loud.

Row one is the only one that costs you anything in the moment, and it is worth every dollar. A stated correction, dated and sourced, is a defensible business decision. A silent swap between message 9 and message 21 is indistinguishable from fraud in a filing — and it's the version your transcript will show.

Step 4: Audit for multi-answer threads

-- threads where the same commitment key got more than one distinct answer
SELECT l.conversation_id,
       l.question_key,
       count(DISTINCT l.answer_text) AS distinct_answers,
       count(*)                       AS times_asked,
       min(l.locked_at)               AS first_answered,
       array_agg(DISTINCT l.answer_source->>'doc_id') AS sources_used
FROM answer_ledger l
WHERE l.locked_at > now() - interval '90 days'
GROUP BY l.conversation_id, l.question_key
HAVING count(DISTINCT l.answer_text) > 1
ORDER BY distinct_answers DESC, times_asked DESC;

At the appliance company this query returned 71 rows on the first run, with sources_used showing {warranty_policy_v4, promo_2023_spring, mfr_parts_sheet} — the entire root cause in one column. Nobody needed a better model. They needed to see that three documents were answering one question.

Check 2 is the preventive one — it catches the conflict before a customer ever sees it. Check 3 is the one that tells you whether the lock is actually wired in or just documented. Check 5 is the fastest way to find answers the model invented outright, and it usually returns a small, fixable list rather than a systemic problem.

The numbers, for the internal pitch

• 71 of 214 repeat-question threads had contradictory answers — 33%. Every one is a written statement the company may have to honor.
• 19 threads had three or more variants. Those are the ones that reach a lawyer.
• One $740 repair settled because of message 9. The legal cost was several times the repair, and the warranty itself was never in dispute.
• Three documents were answering one question: current policy, a 2023 promo page, a manufacturer's parts sheet. The fix was archiving two of them and adding a key registry.
• After the gate: 90 days, 0 multi-answer threads, 412 repeat questions served from the ledger, and average handle time down 9% — replaying a locked answer is cheaper than re-retrieving one.

The thing to internalize is that consistency is not a model property. The same model, at temperature 0, with a good index, will still answer "how long is the warranty" differently on Tuesday than it did on Monday if Tuesday's retrieval lands somewhere else. Consistency is a state property: somebody has to write down what you promised, and then read it back.

A customer asking the same question four times is not being difficult. She is checking whether you mean it. The answer you give her the fourth time is the one that will be read aloud.

If you want the working versions — the answer ledger schema, the eight-branch consistency gate, the correction script table, and the multi-answer audit query I run on every deployment that makes promises in writing — it's here: https://whop.com/agentfast-04b6/