When Is an AI Rewrite Finished?

AI makes it easier to attempt a replacement. The harder question is what the new software still owes the people using the old version.

Imagine rewriting a game engine and getting the same winner in both versions. That looks promising. Then you inspect the match record. The new engine has counted a move supplied by the software as though the AI model chose it. The game still ended, but the experiment changed.

I am exploring a Rust version of eSlams alongside its Python engine. eSlams lets AI models play games so we can investigate their decisions. That gives a rewrite a particular obligation: preserve what a result means. Which moves did the model choose? What could it see? What happened when it failed to return a usable response?

These questions make the appeal of AI-assisted rewrites more interesting to me. A replacement that once looked too expensive may now be worth attempting. We can explore a different language, remove an awkward dependency or reconsider an architecture. But generating the replacement is only part of finding out whether it can take over.

Cloudflare's Vinext offers a useful example. The company described building its initial reimplementation of the Next.js API on Vite in under a week, with one engineer directing AI and about $1,100 in model tokens. It called the release experimental. Seven months later, its account of version 1.0 described the work of making the replacement behave correctly across real applications.

An October security advisory showed how small a consequential difference can look. Two different cached functions could share an entry when they used the same extra cache keys and arguments. In the advisory's example, a public summary route could return data cached by a private receipt route. Vinext 1.2.0 fixed that defect by including the function's source code in the key.

That distinction mattered because the functions had different jobs. A caller could supply the expected arguments, receive a valid response and still get information the route should never have returned. Matching the visible shape of an API had left a gap in its behaviour.

The lesson I take from this is to make the obligations of a rewrite explicit before celebrating its completeness. Some will be documented. Others live in tests, old incidents and the assumptions of whoever calls the software. A particular error may tell another service to retry. The order of two events may determine whether a customer is charged again.

For eSlams, I would want the two engines to encounter the same positions and recorded actions, then compare the state after each move. An identical final score could hide an earlier disagreement. I would also want to know whether they reject the same invalid action and preserve the same information boundaries between players.

The same approach applies elsewhere. Compare a write followed by a read, a request repeated after a timeout, or two customers making otherwise identical requests. Include the stored data and emitted events in the comparison. A response body can look correct while the next request inherits the mistake.

Some differences will be improvements we intended to make. An old bug need not become a permanent feature merely because a comparison found it. But someone needs to decide which behaviour is wanted, understand who depends on it and record that decision. Otherwise, convenient differences can quietly become accepted ones.

Those decisions also affect the release. I would begin with a bounded part of the workload and keep a way back. Running both versions can help, provided the comparison does not send two emails or make two payments. Switching back only works if the old software can still understand the data the new version has written.

I find the possibility of building a better engine exciting. AI gives a small team more room to attempt work it might previously have postponed. That is a meaningful change even before anyone claims a fully autonomous rewrite.

But the finish line has to include the people and programs that depend on the result. They need the replacement to keep working when an input is unusual, a request repeats or a component fails. I would call the rewrite finished when I can explain which promises survived, which changed deliberately and how we checked.