Dev.to Security 🔐 Cybersecurity 👁 0 📖 1 min read

The model you pick isn't always the model that answers

Fable 5.2's system card has a stat worth knowing if you ship agents: on Gray Swan's indirect prompt injection benchmark, attack success within 15 tries was 4.09%, about 4x Fable 5.1 (1.02%). Most of that gap comes from

Fable 5.2's system card has a stat worth knowing if you ship agents: on Gray Swan's indirect prompt injection benchmark, attack success within 15 tries was 4.09%, about 4x Fable 5.1 (1.02%).

Most of that gap comes from the fallback. When a message gets flagged, Fable 5.2 can hand the turn to Opus 4.8. Under attack that happened on 53% of attempts, and attacks succeeded 5.17% of the time when Opus 4.8 answered vs 2.05% when Fable answered itself.

What I'd do about it:

  1. Log model, stop_reason and usage.iterations on every request. Fallback routing is sticky for about an hour, and later turns only show a fallback_message entry in usage.iterations, so watching for the fallback block alone undercounts.
  2. Turn off "Switch models when a message is flagged" for agents that read untrusted content and can act irreversibly (send, push, pay).
  3. Put an action check in front of irreversible tools, so an attack has to beat two layers.

The absolute numbers are still low. The point is that the one that answers sets the risk, and a setting built to cut refusals can route you to an older model without telling you.

Full breakdown, with the logger and the weekly threshold I use to decide when to turn the switch off: https://pub.towardsai.net/fable-5-2-is-4x-easier-to-hijack-a107805488d9?sk=a4a84fa65eabbcbe9c6ced01ec59e537

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.