Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 6 min read

The Random-Bank Control: Why This Memory Ladder Tests "Any Example" First

This article was written by an AI agent from the project's README and release notes, and published automatically. The project is by krapcys1-maker; the code is at krapcys1-maker/adaptive-agent-memory-research. Most wri

This article was written by an AI agent from the project's README and release notes, and published automatically. The project is by krapcys1-maker; the code is at krapcys1-maker/adaptive-agent-memory-research.

Most write-ups about memory for LLM agents compare a memory system against no memory and report the difference. The adaptive-agent-memory-research project takes a different position: before asking whether retrieval finds the right experience, it asks whether showing any experience already does most of the work. That question has a concrete experiment attached to it, and the random-bank control is the piece of the ladder that answers it.

This article is about that one control: what it is, where it sits in the sequence, what it is supposed to separate, and what it cannot tell you on its own.

The problem the control addresses

An agent acting in ALFWorld (a text home simulator) is given a task, takes steps, and either solves it or not. The project's central question is narrow: does showing the agent the right pieces of its earlier successful runs, at the right moment, make it solve more tasks β€” and is it the match that helps, or just showing any example?

The second half of that question is the awkward one. If you retrieve a stored trajectory by similarity to the current state and the agent improves, you have not yet shown that similarity mattered. You may have only shown the agent a worked example, any worked example in the domain, and that alone could shift behaviour. Text in the prompt changes what the model does; the interesting claim is that which text is chosen matters.

Without a control that holds the amount of retrieved text roughly constant and randomises its content, a positive result is unattributable.

Where the random-bank control sits in the ladder

Each rung of the project's ladder adds one thing and is compared on the same tasks, model and budget against the rung below, against no memory, and against the random-bank control. The rungs, as listed in the README:

Rung Variant Question it answers
0 no memory, 3 seeds baseline and noise floor
1 fixed expert examples per task type is a plain example enough?
2 random entries from the bank, every step does showing anything from the domain help?
3 random entries of the same task type is matching task type enough?
4 retrieval by task description, once at the start match to the task
5 retrieval by the current state, every step match to the situation
6+ one improvement per rung chosen from what earlier rungs taught us

The random-bank control is rung 2, and it recurs as the third comparison at every later rung. The ordering is deliberate. Rung 0 establishes how many tasks flip by chance across seeds β€” the noise floor β€” before any treatment is compared. Rung 1 tests whether a hand-picked example is already enough. Rung 2 removes the hand-picking: entries come from the bank at random, every step. Only at rung 3 does matching enter at the coarsest level (same task type), and only at rungs 4 and 5 does similarity to the task or the current state become the retrieval key.

That sequence means a gain at rung 5 has to be read against three earlier measurements: no memory, a fixed example, and random bank entries. If rung 2 already produces most of the gain, then whatever rungs 4 and 5 add is the part attributable to matching β€” and it may be small.

What the control actually holds constant

Two properties make the control informative rather than decorative:

  • Same amount of retrieved text. The README describes the control as "same amount of retrieved text, chosen at random." The point of matching length is to keep prompt size, and therefore cost and context pressure, comparable between the treatment and the control. A retrieval method that wins because it inserts more text has not demonstrated that retrieval found better text.
  • Same bank, same tasks, same model, same budget. The comparison rules in the README require every rung to be compared on the same tasks, model and budget. The control is not run on a different task set or a different checkpoint.

There is also a task-type-matched random arm at rung 3, which separates "any domain example" from "an example of the same kind of task." Two random controls rather than one allow you to see whether coarse matching alone closes part of the gap before content-similarity retrieval is introduced.

Notable design decisions

Randomness is a treatment, not a nuisance. In most evaluation setups you would average over random seeds to reduce variance. Here, random selection is one of the arms, so its variance is part of what is being measured. That is why rung 0 runs 3 seeds and why the noise floor is written down before the treatment rungs are read.

Development on a frozen subset, confirmation untouched. Development happens on a frozen 60-task subset of ALFWorld valid_seen. valid_unseen stays untouched until a single confirmation at the end. The random-bank control therefore runs on the development subset only; the confirmation is a separate, final step.

Decision rules written before each run. The README states that decision rules are written down before a run, raw runs are frozen by SHA-256, and no shortcut that could move the result is allowed (for example, cutting the step limit). A control arm is only useful if the threshold for calling a gain real was fixed in advance.

Per-task gain/loss reading after every rung. The README says every task gained or lost relative to the rung below is read and the reason written down before the next rung is chosen. Aggregate accuracy hides the case where the random bank wins on some tasks and retrieval wins on others.

The trade-off

The ladder is slow by construction. Every rung needs its own arm plus the random control plus the no-memory baseline, on the same task set. The project acknowledges the scale: the research is split into small cells β€” two arms on 20 frozen tasks with one seed, about 2 hours on a 12 GB GPU β€” and many people each running a cell is what lets several directions be tested in parallel.

The payoff is attribution. When memory is reported to help, the random-bank control is what lets you say it was the selection and not the presence of examples. The cost is that you spend a substantial fraction of your compute budget on arms that exist to be subtracted.

Current status and limits

Status as stated in the README (15 November 2026): new direction, nothing measured yet in it. Earlier research lines and their results, including negative findings, are kept in the archive index; the previous README is preserved at a specific commit.

The project also records a discrepancy it has not yet resolved. On its local 4B model, its own feasibility check solved 12 and 14 of 20 tasks without memory, while AdaMEM reports 45.2% for the same model on all 140 seen tasks. The README says resolving that gap is direction D0/D2. Until that is understood, comparisons within the project's own setup are the ones the ladder is designed to support.

Which retrieval key and unit work best is listed as open (DIRECTIONS.md, Q2 and Q6). The random-bank control constrains claims about matching; it does not by itself tell you which matching to use.

Help wanted

The project asks for two kinds of contribution:

  • Lend a GPU. Run a cell: two arms on 20 frozen tasks with one seed, about 2 hours on a 12 GB GPU, on your own machine. How it works and what to submit is in docs/research-map/VOLUNTEER_COMPUTE.md; possible directions are in docs/research-map/DIRECTIONS.md.
  • Without a GPU. Good first tasks are listed under the milestone Experience memory: ladder v1 in the issues. The ones named in the README are the random-bank control arm, a per-task gain/loss table and viewer, the polluted-bank overlay, summaries of papers read only as abstracts, and a ScienceWorld port. Start with CONTRIBUTING.md and docs/00-project/first-contribution-walkthrough.md.

If you want to help with the control described here specifically, the random-bank control arm is listed as a no-GPU first task.

Repository: https://github.com/krapcys1-maker/adaptive-agent-memory-research

Code is Apache-2.0; documentation and data are CC BY 4.0.

Source code, issues and releases: krapcys1-maker/adaptive-agent-memory-research. Feedback and contributions are welcome there.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.