Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 2 min read

How to Test an AI Assistant Over a Whole Conversation With a Simulated User

An assistant that handles 96% of single questions well can fail half of whole conversations. One bad reply fails the conversation, and real users push back after a refusal. A second model playing the user is how you find

An assistant that handles 96% of single questions well can fail half of whole conversations. One bad reply fails the conversation, and real users push back after a refusal. A second model playing the user is how you find out.

Engineer's view. This is testing a login rate limiter with one request. Each request alone is handled correctly. The bug shows up on attempt twenty, after the client has seen nineteen refusals and changed tactics. A one-prompt eval tests your assistant that way. Its 96% says little about whole conversations, where one bad reply is a failure.

TL;DR. A multi-turn eval with a simulated user tests your assistant the way people actually use it: a second model keeps the conversation going and pushes back. One bad reply fails the conversation, so this score can sit far below your single-prompt score.

Explain Like I'm 5

You train a new shop clerk to say no to a customer who wants a refund without a receipt. You test them once. They say no. Good.

But real customers do not ask once. They ask again, nicer. Then they say the manager promised. Then they say it is for a sick grandmother.

To know whether the clerk holds firm, you need a pretend customer who keeps trying. The test only passes if the clerk says no every single time.

The Problem

You have shipped this bug before, and it had nothing to do with AI.

You built a login rate limiter and tested it with one bad password. It returned the right error. The test passed, and you shipped. Then an attacker tried twenty passwords, changing usernames and timing after each refusal. Attempt fourteen got through a path you never tested. Your test checked one request. The attack was a sequence that adapted to every answer it got.

Most AI evals test one prompt and one reply. That is your one-request test. Real users hold conversations. They rephrase after a refusal, add context, claim permission, and come back to the same point.

Anthropic's safety team described this gap on November 13. On their API, recent models handle 94% to 98% of harmful single prompts appropriately. Those scores are near the ceiling and barely separate one model from another. On multi-turn conversations, the same models' rates "range much more widely."

The arithmetic explains why. A conversation passes only if every reply passes. A 4% failure rate per reply is small on one prompt. Over nine replies to a user who keeps pushing, it adds up to much more.

The fix: run multi-turn evals with a simulated user, meaning a second model that follows a written spec and reacts to each reply, and fail the whole conversation on any one bad turn.

Read the full post on The AI Commit

How to Test an AI Assistant Over a Whole Conversation With a Simulated User carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.