Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

How to make an LLM agent test replayable in 3 seconds, with no API key

This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur. Most agent results cannot be checked without paying for the same calls again. Her

This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.

Most agent results cannot be checked without paying for the same calls again. Here is a way to make a test checkable by anyone, with no key and no model, and what it took in this repository.

Four habits

  1. Write the plan first. The predictions and the reading rules were written before any call and locked with a hash, so the page cannot be quietly changed after the results.
  2. Save every reply as it arrives. Each step is stored with the model's exact text, its token counts and its timing, outside the session that made it.
  3. Make the request stream deterministic. The same seed gives the same requests, so the replay can rebuild the true state at every step.
  4. Recompute everything from the replies. The replay script rebuilds the held state step by step from the saved text and stops with an error if one flag differs from what was saved.

What it looks like

With Python 3 and no key:

git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
python3 quickstart_replay.py

In about 3 seconds it prints that the held state was exactly right on 4,000 of 4,000 steps, that the model's replies were exactly the ideal reply 3,954 times in 4,000, and the four-way comparison on the first 400 requests, and it says whether each matches the published figure.

Why bother

A number that anyone can recompute in seconds is easier to trust, and easier to attack, which is the point. If you find a step where the replay disagrees with the page, please open an issue.

What this does not show

Replaying saved replies checks the arithmetic and the logic of the test. It does not prove the model would answer the same way again, and the runs were one sampled run each.

AXI is a layer on the host side by Rajnish Choubey. The repository: https://github.com/gaurshalendra2412/axi-runtime

White paper, "AXI: The Cure for the Cost Crisis in AI" (Rajnish Choubey and Shalendra Gaur): https://doi.org/10.5281/zenodo.23958737

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.