Five days of an autoresearch loop on a card-game bot: the trained network added nothing

I gave an agent five days and one job: make the bots of my trick-taking card game stronger. It had full freedom of method. One rule could not be broken: the meter never moves.

What it delivered in the end was a one-line setting and a negative result. Here is what happened, with the numbers from the agent's own journal.

The setup

The game is a pirate trick-taking game with bidding, four players. The bot already in production is a rule-based bot that plays with a Monte-Carlo search: before each move it simulates possible deals of the hidden cards.

The agent got a mission file. Success was measured as games won (the deciding number) and average points per game (a guard), against that production bot, on 400 deals per measure unless stated otherwise. Every attempt went into a journal: hypothesis, numbers, verdict, reason.

I wrote and rewrote the mission, I stopped the experiment and I ordered the deployment. No human validated individual attempts.

What the agent tried

1. A neural network that guesses the hidden cards. After a reset, the starting network won 47.1% of games and scored 196.3 points per game (reference bot: 197.1). The agent trained a head that predicts where the hidden cards are. One "champion" reached 59.6% and 204.0 points, and held once on fresh deals. Three more rounds of the same recipe all fell back to 49-52%. The 59.6% was mostly a lucky draw: initialisation alone moves results by about 1.6 points.

2. A lever that is not machine learning. More simulated deals per move for its own seat (16 to 48) gave 60.6% and 224.6 points. Mapping win rate against thinking time showed a plateau between 750 and 1000 ms.

3. An isolation test. The rule bot alone at 1000 ms: 64.1% and 236.6 points. The champion with its network at the same budget: 64.3% and 232.6. The trained network added nothing. The research moved to the search itself.

4. A winner that was not one. A sweep of evaluation weights found a variant worth +4.3 points of win rate over 800 deals. Measured again without a time limit, it lost (62.8% against 64.1%). It was an artefact of measuring under a time limit on a machine that could not finish its simulations in time. Thirteen weight variants, three sample budgets, six time budgets and eight training rounds later: no change to the weights, no network.

What shipped

The rule bot with 256 simulated deals per move: 63.8% won and 239.4 points per game over 1,600 deals, against 60.0% and 220.9 over 1,200 deals for the bot that was in production. That is +3.8 points of win rate and +18.5 points per game, from a single setting.

The measurement mistakes the loop caught

Three mistakes in the measurement itself were found on the way:

• Two machines judged with two different builds of the judge for two days.
• 20% of the deals in the production data were duplicates.
• A relative measure ("share of points taken from the current champion") rewarded a champion for beating its predecessor while it was losing to everyone else.

What I changed in the method

The repository has a small runnable version of the loop. It builds in what this run taught me:

• Check the meter first. Four identical reference bots must each win about 25%.
• Measure the initialisation lottery. Train the same recipe twice with two seeds. The margin to keep a change is the larger of 2 points and that lottery.
• Two benches. A fixed bench decides; a control bench is used only to confirm.
• Same training budget for every attempt.
• Keep only if the win gain is above the margin, it is not negative on the control set, and points do not drop by more than 5 per game.

Try it

Python 3.10 or later, standard library only. No model, no API key.

git clone https://github.com/anthohunt/la-boucle-cas-jeu-de-plis && cd la-boucle-cas-jeu-de-plis
python3 -m unittest discover -s tests -t .       # 20 tests, a few seconds
python3 -m trickloop measure --bot rules         # the meter checks itself: about 25%
python3 -m trickloop loop --workdir run --passes 4 --preset fast
python3 -m trickloop dashboard recit/journal.jsonl -o story.html   # the real five-day journal as a page

The fast preset ran in 34 seconds on my desktop (four passes: two changes kept, two thrown away). Its measures are short, so a "kept" verdict can still be luck; the margin and the control set are there to show it.

The game in the repository is a simplified pirate trick-taking game written for it (48 cards, six rounds), not a commercial one. The real experiment's files (mission, method notes, journal, results) are in recit/, in French.

Credits and links

• Case study: la-boucle-cas-jeu-de-plis
• The engine of the method, with a sealed exam the agent never sees: la-boucle-engine
• Inspired by karpathy/autoresearch (no code copied).
• Background on bidding in a close cousin of this game: Cohensius, Meir, Oved, Stern, Bidding in Spades, arXiv:1912.11323.

I'm Anthony Hunt. I build agent systems that run every day and small models that do one job well. More projects: ingevision.cloud.

This post was drafted with an AI assistant from the experiment's journal; every number comes from recit/journal.jsonl and recit/progression.jsonl.