Your Pen Test Passed. Your AI Agent Can Still Be Talked Into Leaking Data

Most security teams have a penetration test on the calendar. Once a year, or before a major release, a team of testers probes the servers, the APIs, the logins, and the network rules, then hands over a report. If the report comes back clean, everyone relaxes.

For traditional software, that confidence is earned. For an AI agent, it can be dangerously misplaced.

Here's why. A penetration test works because the system under test holds still. The code is known. The interfaces are fixed. The same input gives the same output, so a finding can be reproduced, fixed, and verified. The blast radius of a compromised account is defined by its permissions.

An AI agent breaks every one of those assumptions.

The Teller Problem

Picture a bank branch. A pen test checks the vault door, the locks, and the alarm system. It tells you whether a burglar can break in.

What it doesn't tell you is whether the teller will hand over cash to a customer with a convincing story.

In this picture, your AI agent is the teller. It already has the keys. It can read email, query the CRM, issue refunds, and update records. Nobody has to break in, because the agent is already inside. The question is whether someone can talk it into misusing what it holds.

That is a completely different kind of test.

Why Agents Break the Pen Testing Model

Its behavior shifts. An agent's responses depend on the conversation, the documents it retrieves, and the model underneath it. Change any of those and the behavior changes.

Its interface is language. In a normal app, attackers target ports, login forms, and input fields. With an agent, every sentence it reads is a possible attack, including the ones buried inside an email, a support ticket, or a web page.

Its scope is everything it can reach. A pen test scope lists systems and network ranges. An agent's scope includes the system prompt, every tool and its permissions, every document it can retrieve, and every system those tools touch.

Its results are probabilistic. Language models don't behave identically every time. An attack can fail nine times and succeed on the tenth. If an attack works one time in ten, a single test run catches it only 10% of the time. Five runs catch it about 41% of the time. Twenty runs catch it about 88% of the time.

A pen test finding reproduces on demand. An agent finding often doesn't.

This Isn't Theoretical

Microsoft's AI Red Team has tested more than 100 generative AI products. One of its takeaways is that these systems amplify existing security risks and introduce new ones. Their researchers also stress that securing AI is never a finished job.

Then there's EchoLeak (CVE-2025-32711, CVSS 9.3). A single crafted email, with no clicks required, could make Microsoft 365 Copilot leak data. Microsoft fixed it, and there was no evidence of attacks in the wild. But look at the mechanism. No port scan found it. No login test found it. The attack arrived as ordinary content that the agent treated as instructions.

The OWASP Top 10 for Agentic Applications, published in December 2025, puts this class of risk at the top of its list, starting with agent goal hijack and tool misuse.

Pen Testing vs AI Red Teaming

The two disciplines ask different questions, and they are easy to mix up, so it helps to see them next to each other. This breakdown of AI red teaming vs penetration testing lays out the full comparison, including what each approach tests, how it attacks, and how long its results stay valid.

The short version:

• A pen test asks: can an attacker break in?
• AI red teaming asks: can an attacker make the agent cause harm using its own access?
• A pen test checks servers, code, networks, logins, and APIs.
• AI red teaming checks instructions, tool calls, memory, retrieved content, and permissions.
• A pen test finding reproduces every time and stays valid until the code changes.
• An agent finding needs many runs and expires when the model, prompt, tools, or data change.

You need both. Neither replaces the other.

Four Things to Test in Any AI Agent

Once you accept that the agent itself needs testing, the next question is where to start. These four checks cover most of the ground.

1. Goal hijack

Can someone change what the agent is trying to do? An attacker hides an instruction in an email, ticket, or document the agent will read later. If the agent treats that content as part of its task, the attacker now controls it.

Test: does the agent follow instructions that arrive inside content, rather than from the user or the system prompt?

2. Tool misuse

The tool works exactly as built. The agent is the one that gets misled. It deletes instead of archiving, emails an outside address, or issues a refund above its limit. Pay attention to the tool call itself, not just the reply. An agent can say "I can't do that" while its log shows it already did.

Test: does every tool call match what the verified user is allowed to request?

3. Multi-turn pressure

Slow attacks spread a single goal across many harmless-looking messages. In research by Scale AI, human multi-turn jailbreaks succeeded more than 70% of the time against defenses that had reported single-digit success rates for automated single-turn attacks. A login form can't be talked into anything. An agent can.

Test: does the agent still refuse on turn ten what it refused on turn one?

4. Permission boundaries

Ask for another customer's order. Ask a basic user's session to run an admin-only action. The agent holds the keys, so the real question is whether it checks who is asking before using them.

Test: does the agent refuse, or limit the call, when a request goes beyond the verified user's rights?

Why a Yearly Report Isn't Enough

A pen test report stays accurate until the code changes. An agent changes more often, and more quietly. A model upgrade, a prompt edit, a new tool, or a new document source can each reopen a hole you already closed.

That means a yearly report is a snapshot of a system that may no longer exist. The practical fix is to retest after every meaningful change, which is only realistic if you automate it.

The Takeaway

Keep your pen test. It still catches real problems, including classic flaws like server-side request forgery inside AI apps. But add a second test aimed at the agent's behavior:

1. Map everything the agent can reach.
2. Turn each item into an attack goal.
3. Attack with multi-turn conversations, not single prompts.
4. Run each attack several times and score every run.
5. Judge the tool calls, not just the replies.
6. Re-run the suite on every release.

Your infrastructure can be perfectly secure while your agent is perfectly persuadable. Test for both.