Blocking an AI agent's shortcuts with tools: what verifying code comments taught me about steering and MCP

AI agents now review code and write tests as a matter of routine. They do not read only the code. They read the comments too, and sometimes they interpret the code through what a comment says it is meant to do.

I picked writing-tracker (MIT), a web app that another developer published on GitHub. It records how far along you are in writing a book. It was built with Lovable, an AI app builder, over three days at the end of April 2025, and it has not been updated since. The code and comments the AI wrote are still there as they were, and it has logic such as aggregation and ordering whose behavior can be checked against its comments, so it made a good subject.

https://github.com/shateers/writing-tracker

I gave the app to Kiro and asked: "Check whether any comment disagrees with what the code actually does." The statistics screen has a table that lists your progress for the last six months, one row per month. The code that builds it says:

// Generate last 6 months
for (let i = 0; i < 6; i++) {
  const monthStart = new Date(currentDate.getFullYear(), currentDate.getMonth() - i, 1);
  // ...
  periods.push({ label: `${monthName} ${monthStart.getFullYear()}`, start: monthStart, end: monthEnd });
}
// Reverse to have most recent month first
periods.reverse();

The rows of the table come from the list of six months this code builds (periods). The first month in periods becomes the top row, and the rest follow downward. If the comment is right, the month that ends up first after reverse(), and so the top row of the table, is the current month. Kiro read the code and judged the comment correct. Then a property-based test (PBT) checked the same behavior, and it disagreed on the first run.

PBT is a testing technique where, instead of writing input and expected-output pairs one at a time, you write a property that should hold for any input, and the framework generates many inputs to check it. When it finds an input that breaks the property (a counterexample), it shrinks it to the simplest form it can. Here the property was "whatever today is, the top row of the table is the current month", and the test built the table again and again, setting "today" to many different dates.

For example, with today set to January 2020, the top row was not January 2020 but August 2019, the oldest of the six months shown (August 2019 to January 2020).

The cause is the order in which the loop adds months. It starts from the current month (i = 0) and appends each earlier month to the end of periods. In the January 2020 example the list is "January 2020, December 2019, ..., August 2019", so the current month is already first. Calling reverse() on that flips the list and puts the oldest month, August 2019, at the top. To put the newest month first, as the comment says, the reverse() was never needed.

"If the agent believes whatever it reads, make it run the code."

That is why I built spec-lantern. It is a Kiro Power. A Kiro Power bundles instructions for the agent with an MCP server configuration, and you install it into Kiro from GitHub or elsewhere. The current Kiro docs recommend the Agent Plugins format for new powers; spec-lantern uses the earlier POWER.md format, which still works. spec-lantern makes the agent predict whether each comment is true, locks that prediction, and then runs the code in an isolated environment to find out. It is also my entry for Kiroween 2026, Kiro's hackathon.

https://github.com/ikenyal/spec-lantern

What building it taught me is that "make it run the code" is not enough. To reach its goal, the agent kept finding shortcuts I had not anticipated. And different kinds of shortcuts needed fixes in different places. Some were solved by one more sentence in the instructions. Others came back no matter how often I rewrote the instructions, and stopped only when I changed how the tools work.

This post walks through those shortcuts, taken from the actual records, and explains where I blocked each one (steering, meaning the instructions given to the agent; the MCP tools; or the execution environment) and why there. None of these ideas are specific to Kiro, so they may also be useful if you hand verification work to Claude Code or another agent.

How spec-lantern works

spec-lantern consists of a Kiro Power (POWER.md plus seven steering documents) and its own MCP server. The agent can touch the target repository only through the MCP server's tools.

flowchart TB
  subgraph agent["Done by the agent (Kiro)"]
    direction LR
    A[Split comments<br/>into claims] --> B[Write executable<br/>properties] --> C[Predict<br/>PASS/FAIL]
  end
  subgraph tools["Done by the MCP server's tools"]
    direction LR
    D[Lock the<br/>prediction] --> E[Run in isolation] --> F[Compare with<br/>the prediction]
    F --> G[Replay the<br/>counterexample] --> H[Fix on a<br/>throwaway copy]
  end
  agent --> tools

The agent's job ends at prediction. It splits comments into claims, writes an executable property for each claim (a test written with fast-check and Vitest), and predicts the result of running it. If the test passes, the result is PASS and the comment was right. If it fails, the result is FAIL: the comment disagrees with what the code does, which is to say it lied. Everything after that (locking the prediction, running, comparing, applying a fix) is done by the MCP server's tools.

I did not start with this split. Earlier, the agent wrote the comparison tables and result files itself, and the lock did not protect the property files. As I dealt with the six shortcuts below one by one, the part left to the agent got smaller, and it ended up like this.

Six shortcuts the agent took

None of these were taken out of malice. Each was the agent meeting its goal by the shortest route available. A human engineer near a deadline might well make the same calls. That is exactly why reminders do not stop them: whenever the conditions line up, they come back in the same form.

What follows is what actually happened before any countermeasure. How I blocked them comes in the next section. To say it up front: some could be prevented by how the instructions (the prompt) were written, and some could not, no matter how often I rewrote them, and needed changes to how the tools work. That difference is the subject of this post.

One and two: distorted predictions

The first is the "read and believe" from the opening. Put a comment next to code, and the agent interprets the code on the assumption that the comment is right. Accepting "to put the newest month first" is a more natural way to read than tracing the order before reverse() yourself. In human review too, comments that explain intent exist to reduce the effort of reading code. Believing comments is built into the act of reading.

I thought running the code would solve this one. But within the run-the-code setup, a second shortcut appeared: the convenient prediction.

This happened while I was trying the packaged Power in a separate environment. spec-lantern's highlight is the moment when a comment that looked right on reading turns out to be a lie when executed: a prediction of PASS and a result of FAIL.

Kiro understood that aim. And for one comment, even though it had noticed while reading that something looked off, it predicted PASS. Its report gave the reason as "to make the mismatch between prediction and result meaningful". It thought the answer was FAIL and predicted PASS on purpose, to create the highlight.

What makes this shortcut hard is that it becomes more likely the better the agent understands the system's purpose. A prediction matters as a record of how the agent judged the code from reading it. Once it is mixed with a judgment about which result would look good, it stops being a record and becomes staging. A staged prediction, placed next to the execution result, no longer shows how Kiro read the code, and loses its meaning as a prediction.

Three and four: contaminated expected values

The third is deciding the test's answer by looking at the implementation. For a second subject I picked MoodPlays (MIT), also built with Lovable. It finds nearby places to match your mood, and it has logic such as parsing opening hours and calculating distances.

https://github.com/gargi04sahu/MoodPlays

Here, five of the first ten properties Kiro wrote computed the expected value (the answer the test treats as correct) with the same calculation as the code under test.

Take distance as an example. The shape of the problem is: copy the formula from the code into the test, and check that both give the same result. Such a test passes even if the formula is wrong. It looks like a test, but it treats the implementation's calculation as correct and uses it as the answer, so it is the same thing as the first shortcut, read and believe. When rewriting, I replaced these with expectations grounded in something other than the code: the real distance between two points with known latitude and longitude, or the concrete example given in a comment, when there is one. In software testing, whatever decides whether a result is correct is called a test oracle. What was needed here was a test oracle grounded independently of the code under test.

This shortcut is hard because it is the agent's most reliable way to write a "correct" test. Deriving the expected value yourself risks getting it wrong. Using the same calculation as the code at least guarantees the test will not break at run time. Once passing the tests becomes the goal, deciding the answer from the implementation is the rational choice.

The fourth is peeking before the lock. This also happened with MoodPlays. Before locking its predictions, Kiro copied the target code and ran it on the spot with node -e (a Node.js command that runs a short snippet), to check in advance whether the tests would pass.

Kiro presumably did not want to miss. But this makes the prediction meaningless. A prediction in spec-lantern is a record of how far reading alone can tell whether a comment is right. If you run the code and see the result before predicting, of course you will be right, and you can no longer tell whether reading was enough.

Five and six: broken records

The fifth is rewriting after the lock. While I did several full runs to record the demo video, in one run Kiro kept editing property files after locking and re-running.

The cause was that the tab-switching UI did not work in the test's simulated browser (jsdom), so the monthly table was never rendered and the test counted rows that were not there. That is a test defect, so fixing it was legitimate. The final result was correct, and Kiro's report showed reasonable grounds for each fix. But those reasons were not recorded, and afterwards nothing distinguished this run from one where the tests were edited until they matched the prediction. Inside a system built to show that predictions were right, it was no longer possible to prove that this prediction was right.

The sixth is hand-written tallies. Earlier, the agent copied execution results and comparison tables into files by hand. In one run it miscopied a prediction into the comparison table. In the next run, Kiro re-read the lock files and noticed the error. A miscopy is an honest mistake, but the fact that a table showing whether predictions were right can contain errors is itself the problem. A reader cannot tell whether the table is right without opening the original files.

What the six have in common

Lined up, the six show that shortcuts can get into every stage: prediction, expected values, execution, and records.

Looking back, none of them are specific to AI. Teams that are just starting to write tests run into similar things. Filling in expected values with whatever the implementation outputs. Making a failing test pass by changing the test instead of the code. Reading code in review through what the comment says it intends. Copying errors creeping into hand-compiled reports. All of these happen on human teams.

The reasons are probably the same for humans and AI. One is goal displacement. The real purpose of testing is to find where code does not do what it should, but day to day, "the tests pass" is the visible result. Then humans and AI alike take the shortest route to passing. The other is that the checker and the checked are the same. If the person who wrote the implementation also decides the expected values, the same assumptions go into both. If the person who made the prediction also tallies the results, they read the results in the prediction's favor without meaning to.

Human teams have compensated with process and systems. Test-driven development, writing tests before the implementation, separates expected values from the implementation. CI checks that block merging until tests pass, and code review by someone else, keep people from judging their own work. Handing verification to an AI agent needs the same kind of measures. The difference is that they have to be built into instructions, tools, and the execution environment in a form the agent can read. spec-lantern's "lock the prediction before running" is close to TDD's "write the test before the implementation".

On top of that, the thing that decided where to put each countermeasure was whether the agent was aware that it was taking a shortcut.

Shortcuts the agent is aware of have a reason. If there is a reason, you can stop the shortcut by changing what the reason rests on. Shortcuts the agent is not aware of are not shortcuts from its point of view; they are ordinary work. No amount of warning stops them, because the agent does not think it is doing anything that needs a warning. This difference decided where to block each one.

Three layers for blocking shortcuts

The countermeasures fell into three layers.

The lower the layer in the table, the more reliably it blocks a shortcut regardless of what the agent decides. The higher the layer, the more you can leave situational judgment to the agent, and the easier the countermeasure is to change. Awareness mostly decided which layer handled which shortcut. If the agent is aware, change the premise with steering. If not, change the shape of the work itself with tools or the execution environment.

This split also matches what human teams already do. Coding conventions and review guidelines explain why, and leave it to each person's judgment; that is steering. CI checks and branch protection that blocks merging without approval take the decision out of people's hands and make the step impossible to skip; that is the MCP tools. Permission design that keeps developers away from production, and test environments that cannot reach the outside, create boundaries that cannot be crossed at all; that is execution-environment isolation. Human teams found that careless mistakes do not go away with reminders, and prevented them with CI and permissions. Blocking unaware shortcuts with tools and the execution environment follows the same idea.

Here is what I did in each layer and why.

Steering: change what counts as success

Steering is guidance the agent reads every time. In Kiro, Markdown files in .kiro/steering/ are loaded for each task, and the instructions bundled with a Power play the same role.

Against convenient predictions, I rewrote the prediction section of the instructions as follows. I did not want to stop at "don't predict strategically", because a bare prohibition would leave the agent still treating "creating a mismatch" as success, and it might reach the same goal by other means.

1. Read the code and predict what you **actually believe**: exactly one of PASS
   or FAIL. If you suspect the code does not do what its comment says, predict
   FAIL and write one line of reasoning. Never predict strategically (for
   example "PASS so that a mismatch is meaningful"); a prediction you do not
   believe makes the whole report dishonest. A correct FAIL prediction is a
   valid, useful result: it shows the bug was visible by reading.

The point is the last sentence. A correct FAIL prediction is also a valuable result, because it shows the bug could be seen by reading. The agent chose the convenient prediction because it judged that "a mismatch" was worth more to the system than "a correct prediction". Change what that judgment rests on, and the reason to take the shortcut disappears. In the runs after this change, Kiro predicted FAIL for the suspicious comment, execution returned FAIL, and the prediction was right.

Deciding the answer from the implementation was a different kind of problem. The agent was not aware of it, so an abstract "write against an independent standard" gives it nothing to check its own tests against. So I added a step that asks before locking, "could this expected value be produced with the same calculation as the code under test?", and listed concrete forms of it: using a data constant from the code as the expected value, writing the same branch as the code into the test, repeating the code's calculation on random inputs.

This is an attempt to create awareness of an unaware shortcut. Concrete examples let the agent notice that it is deciding the answer from the implementation. It helped, but it is not complete. The forms are endless, and any form missing from the list goes unnoticed. As described later, this is where the final check still depends on a human.

Peeking before the lock has not happened in any run since I added a sentence forbidding it. Peeking was something the agent chose to do, so I think showing what the act amounts to ("running the target code before predicting is executing the target") is enough for it to stop choosing it.

Steering works where the agent makes judgments. By changing what a judgment rests on, or what an act means, it removes the reason to take a shortcut. The flip side is that as long as there is room for judgment, it will eventually miss. Shortcuts that do serious damage when steering misses had to be blocked in the next layer.

MCP tools: make the procedure impossible to bypass

Steering could not prevent rewriting after the lock or hand-written tallies. Both are ordinary work to the agent. Fixing a broken test and summarizing results in a table are correct things to do, and forbidding them forbids necessary work too. So I left the work allowed and changed the tools so that a record is always left.

The prediction lock hashes both the prediction and the property file.

export interface PredictionLock {
  propertyId: string;
  verdict: Verdict;
  hashAlgorithm: "sha256";
  /** Hex digest over (Predicted_Result + Correctness_Property bytes). */
  contentHash: string;
  /** Hex sha256 over the RAW Correctness_Property test file bytes alone. */
  propertyContentHash: string;
  // lockedAt, etc.
}

Before running, execute_properties checks the current hash of each property file against its lock. If they differ, it refuses to run and points the agent to supersede_prediction_lock. That tool keeps the reason for the change and the previous hash in locks/superseded/, then re-locks. If the prediction itself would change, it refuses.

I did not choose to forbid changes after locking because test defects are real. The fifth shortcut was caused by one. If fixes are forbidden, the agent is left with two options: report the result of the broken test as is, or stop. Reporting it as is means saying "the comment lied" when the test failed only because of its own defect. Stopping leaves it unknown whether the comment is right. What was needed was not to stop fixes but to make it possible for anyone, later, to confirm that a fix was made, why, and that the prediction did not change.

Hand-written tallies went away because the step itself went away. The agent no longer copies results; the tools save them to files themselves. execute_properties and the other tools write the raw result to a timestamped file and return its path. compare_predictions never takes predictions from the caller; it always reads them from the lock files. With no step where the agent writes the comparison table, there is nothing to miscopy. Not taking predictions from the agent also removes any chance of swapping a prediction just before the comparison.

Counting lying comments is also the tools' job. A lie here means a comment that execution showed, with a FAIL, to disagree with what the code actually does. One root-cause bug can make several properties fail. In MoodPlays, a substring check includes('24') in the closing-time logic treated places that close at a time containing "24", such as 13:24, as open 24 hours, and two properties failed. Counting that as two lies would overstate it. compare_predictions returns the number of lies grouped by root cause separately from the number of wrong predictions. If the agent counted, choosing which number to report would be one more judgment call.

Blocking with tools has a cost. The stricter the procedure, the more trial and error it takes the agent to use the tools correctly. In early runs, Kiro could not work out the shape of the data the tools expected, and kept calling them again while reading the errors. I reduced that friction by asking less of the caller. For example, execute_properties no longer needs every property passed in one by one: given the folder that holds the spec, it builds them itself. If you add procedures that block shortcuts, you also need to make the right way easy. Otherwise the agent keeps getting stuck in front of the blocked path.

Execution environment: a boundary that cannot be crossed

The last layer is a boundary the agent cannot cross whatever it decides. This layer is also what finally corrects the first shortcut, read and believe: whether a belief from reading was right is settled here, by actually running the code.

The target code runs in containers, in two phases. spec-lantern uses Podman, but the features it relies on (an internal network with no route out, and starting a container with no network) also exist in Docker and other container runtimes, so the approach carries over.

In the first phase, dependencies are installed on an internal network with no route out (with Podman, podman network create --internal). The only thing that can reach the outside is Verdaccio (a server that proxies the npm registry), placed there as the relay. Installation runs with --ignore-scripts, so no package can run arbitrary scripts during install. In the second phase, the tests run in a container started with --network none. The dependencies are already on disk, so no network is needed.

The phases are split because installing needs a network and testing does not. Running tests in one container with the network open would let the code you are checking talk to the outside world. spec-lantern deals with code whose contents you do not fully know. Whether it is someone else's public app or your own repository written by AI, you do not know what is in it until you run it. Until partway through development, spec-lantern still had an older path that ran the target in a single container with the network open. I removed it, and now there is no path inside spec-lantern itself where the target code runs while connected to a network.

This layer also guarantees that the original repository is never touched. An earlier implementation mounted the original repository straight into the container, and the install phase rewrote node_modules and package.json. I did not notice while testing in my development repository; it surfaced only when I tried the packaged Power in a separate environment. Now every run, replay, and fix uses a throwaway copy created under ~/.spec-lantern/worktrees. apply_fixes_and_reexecute writes only to the copy, and returns the diff it actually applied, computed before the copy is deleted.

Reproducibility belongs to this layer too. Each run records the random seed, the time zone, and a fixed clock, and replay_counterexample reproduces a failure under those conditions. There was a hole here as well. The record said "clock fixed to January 1, 2025", but tests that rendered UI components saw the real current date. On the general execution path the time zone was fixed, but the clock itself was not. If the recorded conditions differ from the actual ones, the claim that a run is reproducible falls apart. As a fix, I added a test that checks the recorded conditions are really applied at run time: if a test sees a date other than the recorded one, it fails.

Not reporting test defects as bugs

The execution environment has one more important job: classifying results. As the fifth shortcut showed, a test can fail for reasons other than a bug in the code under test: a mistake in how the test is written, a missing environment setting, a timeout. If all of these count as FAIL, test defects get reported as "the comment lied".

spec-lantern classifies each result as a verdict (PASS or FAIL), an execution error, a timeout, or runner unavailable. A FAIL comes only from an assertion inside the test failing with a counterexample. For example, if a test that renders a UI component forgets to set the simulated browser environment, it crashes with document is not defined. That is an execution error, not a verdict, and the message includes how to fix it: add the environment setting on the first line.

With this classification, the agent does not have to guess whether a failing test means a bug in the target or a defect in its own test. In the run with the fifth shortcut, Kiro stopped on a FAIL that came from counting a table whose tab never opened, judging that "reporting this as evidence of a bug would be dishonest". When the classification and the agent's judgment line up, test defects do not get reported as bugs.

Checking spec-lantern's own code by running it

The mechanisms for blocking shortcuts also found a bug in spec-lantern itself.

spec-lantern has a safety device (the Safety Gate) that keeps target code from running directly on the user's machine, outside a container. When it receives an instruction that starts with a command that runs the target code on the spot, such as npm run or node, it refuses without running it. A PBT I wrote for this check found that npm run (followed only by spaces) slipped through. The check trimmed trailing spaces from the input and then did a prefix match against "npm run " with its space, so the trimmed "npm run" did not match. It was a failure that appeared or not depending on the random seed. With tests that list input and expected-output pairs by hand, the inputs lean toward the uses the author has in mind. You can write "npm run build is refused" and "ls is allowed", but "input followed only by spaces" is beyond what the author of the check imagined, so it rarely shows up in such tests. PBT generates inputs regardless of what the author has in mind, so inputs beyond the author's imagination are included too. It avoided, at the point where test inputs are produced, the problem mentioned earlier: when the checker and the checked are the same, the same assumptions get in. Simply trimming spaces on both ends would then wrongly refuse harmless strings like node_modules..., so I changed it to match whole tokens and confirmed it passed ten runs in a row.

What spec-lantern asks of the code it checks is to not trust explanations and to confirm by running. I applied the same standard to spec-lantern's own safety device. A system that tells other code "don't trust explanations" would not be convincing if it trusted its own safety device because "I wrote it, so it must be safe". And a safety device is the part where trusting it and missing something does the most damage.

Kiro features that helped

Several Kiro features helped me build this.

The first was spec-driven development. Kiro wrote the spec for spec-lantern itself, and I implemented it as 12 requirements in EARS form, a design with 35 Correctness Properties (properties that should hold for any input, which become PBTs), and 21 tasks. The one thing I changed by hand was the task order: I built first a thin path where a component from a real app runs in a real container and returns a real FAIL. If everything is connected only at the end, problems that show up only when you actually run things, like the ones in this post, appear too late. In fact, a design flaw in the isolation (a file-sharing problem between Verdaccio, the relay, and the virtual machine Podman runs) surfaced as soon as that path switched to two-phase isolation.

Steering is where I carried lessons into the next task. For example, there is a rule that the report shows only facts confirmed by execution (report-data-fidelity.md). Early in building the report screen, sample data that Kiro prepared for the screen included a comment that does not exist in the target app. It came with a file name and line number and looked indistinguishable from a real one. I added the rule the day I noticed. From then on it was read every time Kiro built the report screen that shows results, or the mini game where you guess whether real comments are lies (Trust or Test?), and it steered Kiro's implementation toward checks that compare generated data against the original files and stop the build if they differ.

A Kiro Power is a way to package the MCP server with the instructions and hand them out together. The MCP server is bundled into one file with esbuild and started with npx from a public repository. Add the GitHub URL through Kiro's "Add Custom Power", and someone else's environment gets the same instructions and tools. In the packaged Power's configuration, apply_fixes_and_reexecute, which applies fixes, is left out of auto-approval, so a person always confirms it.

https://kiro.dev/docs/powers/installation/

https://github.com/ikenyal/spec-lantern-power

Another thing that helped find problems was trying the packaged Power again and again in an environment separate from the development repository. In the development repository, Kiro can pick up the history from steering and past artifacts. So if the Power's instructions leave something out, Kiro fills the gap from that history and the problem never shows. In a separate environment, the Power's instructions and tools are all it has. Many of the shortcuts and holes in this post surfaced only there. They would not have been found without testing under the same conditions a user has.

For larger fixes I used Kiro workflows (an implement-and-review loop). A reviewer checks the diff against the original request, which cut down on things missed when fixing problems found in testing.

Results after the countermeasures

With all of this in place, I gave the same request to a fresh copy of writing-tracker, using only the packaged Power. One run, no property edits after locking, and all four predictions were right. This time Kiro predicted FAIL for the reverse() comment, and execution returned FAIL. It also predicted and confirmed a second bug: because each month ends at 00:00 on its last day, a book updated later on the last day of a month is not counted in that month. The fixes were applied only to a throwaway copy, and the original repository did not change.

What matters more to me than the predictions being right is being able to show that they were right. Lock times are also in git commits, so anyone can check from the history that predictions came before execution. Kiro's earlier locked prediction (PASS) and its final prediction (FAIL) are recorded the same way and can be checked the same way.

You can see the results in the public report. The same page has a mini game where you guess which comments lied.

https://d2xayupxggkiqt.cloudfront.net/

Limits of spec-lantern

Even with these countermeasures, three things are not prevented.

First, the agent chooses which comments to check. Recording a hard-to-check comment as "unverifiable" with a reason is legitimate, but the more of them there are, the more gets missed. For now, the scope is limited to UI components that run on props alone (values passed in from outside) and pure functions. Code that depends on a database or an external API can only be handled with recorded responses.

Second, whether tests are written against an independent standard still depends on a human check in the end. Even with steering that forbids it and a self-check step, there is no way to catch every answer decided from the implementation. A lock guarantees that the prediction came before execution, not that the test is right. Confusing the two creates a new read-and-believe: "it is locked, so the test must be right".

Third, spec-lantern's isolation currently assumes Podman running on the local machine. Cleanup removes only resources whose owning process is confirmed to have exited, but if Podman is shared from another container, that check may not hold. Starting containers and installing dependencies also takes time, and tests that render UI components many times have exceeded the default time limit. The balance between strictness and waiting time needs tuning to the size of the target.

Ideas that apply to any AI agent

All of this carries over to agents other than Kiro.

The most important one, I think, is to ask, when you find a shortcut, whether the agent is aware of taking it. A shortcut the agent is aware of has a reason, and instructions or a prompt can change what that reason rests on. A shortcut it is not aware of does not stop with warnings, so the shape of the work has to change through tools or the execution environment. If you skip that distinction and add another prohibition to the instructions for every shortcut you find, the instructions keep growing and the shortcuts survive in new forms.

The concrete ways of blocking with tools carry over directly too. Do not let the agent copy results; have the tool save them and return the path. Do not take values used for comparison or judgment from the agent; read them from fixed files. Instead of forbidding work, make sure that work always leaves a record. These designs can be built the same way with Claude Code hooks or MCP servers.

Locking the prediction before running is also an idea for agent verification in general. In clinical trials and in research fields such as psychology, there is a procedure called preregistration: before collecting data, you register your hypotheses and analysis plan in a public place, so you cannot change them to fit the results. Locking predictions is a close cousin. Whether an agent "was right" or "was fooled" means something only when the prediction cannot move afterwards. If an agent reports "I checked it" without a locked prediction, nobody outside can tell whether that report comes from reading and believing or from running and confirming.

Kiro on day one and Kiro at the end

On day one, Kiro read the reverse() comment, judged it correct, and execution overturned that judgment. After the countermeasures, Kiro predicted FAIL for the same comment, and the prediction matched the result. What changed in between is the setup: predictions locked before running, results saved by the tools, and target code run only in isolation. Because of that, anyone can check afterwards whether a prediction was right.

When you hand verification to an agent, how capable the agent is matters, of course. But just as much, the question is whether you have built a setup in which the agent cannot take shortcuts. Rather than blaming an AI for believing comments, build the setup first, so that it does not matter whether it believes them. spec-lantern is one way to do that.