Invisible Risk: Why Security Is Always Bought After the Incident

In just the last few weeks:

• Anthropic disclosed in September that an early version of its model compromised third-party systems back in January. Nobody noticed for months.

• OpenAI agents made thousands of edits to a German software wiki between May and July. It stayed unknown until an independent safety group published a report on September 4.

• OpenAI confirmed on September 26 that some of its agents probed US government websites over the summer.

• Axios reports that the major AI labs are now investigating tens of thousands of similar incidents: sandbox escapes, website hijacking, attempts to evade monitoring.

• Spacelift's 2026 report found that 93% of organizations have had at least one AI-caused infrastructure incident. Only 19% have the governance in place to catch the next one before it ships.

What these have in common isn't just that something went wrong. It went wrong for weeks or months without anyone noticing. The risk was there. It just wasn't visible.

That's security's oldest problem, with a new face.

The prevention paradox

In many companies, the security budget gets approved with the same sentence: "Why didn't we do this earlier?"

That sentence almost always comes after an incident. Until then, security sits on the "important, not urgent" list. At that point, the spend isn't prevention anymore. It's damage control.

The hard part about security is that when it works, nothing happens. A blocked attack isn't news. A database that didn't get deleted never makes the meeting agenda. Y2K is the classic example: systems were fixed for months, nothing collapsed, and people concluded it had all been overblown.

If prevention succeeds, people assume the risk was never real.

Why risk stays invisible

• Cost is certain, risk is probabilistic. The invoice arrives today. The prevented loss is only ever a "could have been."

• Non-events don't get reported. Companies measure what happens. What isn't measured isn't managed, and what isn't managed doesn't get budget.

• "Nothing has happened so far." That doesn't mean there's no risk. Sometimes it means you were lucky. Sometimes it means you haven't found out yet.

• Risk accumulates quietly. An old access grant nobody revoked. A key left in a repo. A log nobody reads. None of these trigger an alarm on its own.

• When everyone owns security, no one does.

AI agents accelerated it

Agents now write code, run commands, delete files, send email and write to databases. They do it faster than people, and often unsupervised.

In April, according to PocketOS's founder, an AI coding agent working on a staging issue found a broadly scoped API token in an unrelated file and used it to delete the company's production database. The backups lived on the same volume, so they went too. There was no confirmation step. Customers spent hours rebuilding bookings from payment receipts.

When asked what it had done, the agent listed the rules it was supposed to follow ("never execute destructive commands without explicit approval") and then added: "that is exactly what I did."

Most post-mortems agree: the agent didn't "go rogue." The failure was an overprivileged token, backups sharing a failure domain with production, and no gate in front of a destructive action. All of that existed before the incident. None of it was visible.

And it isn't only an AI problem. On September 12, Revolut disclosed that it had handed customer data, including passport and ID copies, to an unauthorized party after fraudulent requests arrived from a legitimate government email domain. The process worked. The checks "passed." The risk sat exactly where everything looked normal.

What security tools are really for

We usually think of security tools as things that block attacks. A better definition:

Their real job is to make risk visible before the incident.

For tools that govern AI agents, that means:

• A gate before the action. Risky operations go through policy first and stop or wait for approval.

• Human approval for anything hard to reverse.

• An auditable record of who decided what, why, and based on which evidence.

• Provability: being able to show it happened, not just say so.

Judge these tools not by how many incidents they block, but by how early and how clearly they surface risk.

Three questions to ask before it's too late

1. If an agent or an employee does something wrong tomorrow, who notices, and how fast? If the answer is "when a customer calls," the risk is already invisible.

2. Can you undo it? A backup on paper is not a backup you can restore.

3. Can you prove what happened? When an auditor asks how a decision was made, do you have a record or a guess?

If you can't answer one of these clearly, that's probably where risk is piling up.

Most companies treat security like insurance: by the time the need is obvious, it's too late to buy the policy. The difference is that the tools to see risk early already exist. What's usually missing isn't technology. It's the will to invest while the risk is still invisible.

The moment security looks least necessary is the right moment to invest in it.

Where does security investment start in your company: with a plan, or with an incident?