Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Agents forget, so I write it down

In September I asked a simple question about our system: if I change this table, what breaks? There was nowhere to look it up. We had eleven repositories. One of them had real documentation, and that documentation wasn'

Agents forget, so I write it down

In September I asked a simple question about our system: if I change this table, what breaks?

There was nowhere to look it up. We had eleven repositories. One of them had real documentation,
and that documentation wasn't even in version control. Another one's README was a single line with
its own name. None of them had a rules file telling a coding agent where to start. So every time
the question came up, someone had to read the code again from the beginning. Once agents were
doing the reading, it got no better: a new session didn't have what the last one had worked out,
so it did it all over again.

That's the lesson of this post: records are the agent's memory. Don't assume the next session
has the context from this one. Give it durable records, and check that they are actually loaded.

This is part 5 of a series on what I learned putting AI agents to work in my own job.
Part 4 was about giving an agent the right context for a task.
This one is about what lasts after the task is done.

Records are the agent's memory

Treat a new session like a very capable new hire who missed yesterday. Unless you hand it records,
it knows the world in general and little about your system: not why a table is named oddly, not which status column
is the real one, not the trap you fell into last week. What it does well is read. Written records
give the next session something to retrieve instead of rediscovering it.

So the question isn't "how do I make the agent remember?" It's "what did we write down?"

Records are the agent's memory: do the work, write it down, the next session starts from the records, fewer repeats

Each turn of that loop makes the next one cheaper. Skip the "write it down" step and the loop
breaks: each session has to rediscover what the last one learned, and so does every new teammate.

A documentation hub in a day

On 17 September I decided where the documentation should live. I wrote down three options:

  • Fill in each repository's own docs. Close to the code, but there's no place for the most valuable facts, the ones about how services talk to each other. "This API calls that one" is only half true in either repository, so it would be written twice and drift.
  • A wiki. Easy to edit, but separate from the code, so it goes stale quickly. It would also need a separate retrieval path; a repository was readable with the file tools my agents already used.
  • A separate documentation repository, a hub. One home for cross-service facts, readable by agents with the same tools they use on code. This is what I chose.

The hub holds what belongs to no single repository: how services connect, the database,
operations, and decisions. Docs that live next to the code, like how to run it or an API spec,
stay in their repositories. Then I wrote it, with an agent reading the code alongside me: 63
documents on that first day, plus the first rules files pointing agents at the hub. It has grown
to more than a hundred documents since.

Two decisions from that day turned out to matter more than I expected:

  • Reuse what people already knew. Someone had written an excellent database reference in a repository of operational scripts, not where a developer would look. Its best parts were traps, like which of two similar table names is the real one, or which column holds the true status. You can't get that from reading the code. I copied it into the hub and declared the hub the original from then on, so there'd be one source of truth instead of two copies drifting apart.
  • Split big files. That reference was one 895-line document. I split it by domain, because when a file is huge, an agent reads the parts it doesn't need along with the part it does.

Keeping the docs honest

A documentation hub has an obvious weakness: it's not the code, so it can go stale. Four things
counter that.

How the docs stay honest: each doc names its sources, each repository has a rules file, the Slack agent reports mismatches for a person to fix, and for current behaviour the code wins over an unverified doc

  • Each doc names its sources. Its header lists the code paths it describes and the date it was last verified. A small lint compares that date with the latest change to those paths and flags the doc as stale.
  • Each repository has a rules file. It says: if you change the structure, update the doc. I put the rule where agents read first, rather than relying on anyone remembering.
  • Agents report mismatches back. When the Slack agent answers a question and finds the doc disagrees with the code, it doesn't argue with the person asking. It logs one line, and a scheduled job sends it to me. The Slack agent can't edit the docs; it reports mismatches for me to resolve, and I fix the doc.
  • For what the system currently does, code wins. The rules say that when a doc and the code disagree, the code is right about current behaviour, so an agent doesn't trust an unverified doc over the code in front of it. Whether that behaviour is correct is a separate decision.

Two of these failed in instructive ways:

  • Freshness depends on the refresh path. For the first few days the agents' machine didn't pull the docs hub at all. "Answer from the latest docs" means nothing if the latest docs never arrive. Once I noticed, I added the hub to the list of repositories it keeps up to date.
  • A report can disappear. One mismatch report was blocked by an over-eager safety check: a full stop inside a quoted sentence looked like a file path. The agent didn't retry, and the report never reached me. I found it three days later while reviewing what the guards had blocked, and fixed both the check and the doc. I wrote about that review in the first post on this blog.

From grep to one query

The documentation answered "what is this?" well. "What breaks if I change this?" was still slow.
The agent went from the hub to the code paths a doc named, then searched the code. It missed
things, and the more code it opened, the more chance there was of quoting code back into Slack.

At the end of October I connected the agents to a metadata graph built from the database schema,
the code, the routes, the scheduled jobs and the docs. Now the rule is: ask the graph first, and
open only the files and lines it points to. Questions like "which code writes this table?" or
"what might be affected if this value changes?" take one or two queries, and the answer comes with
file and line ranges as evidence.

It has costs, and they're written down too: the graph is rebuilt by hand, so it can lag behind the
code; some older code builds its queries in ways the extractor can't follow; and impact estimates
come out broad, so the agent says "possible impact", not "impact".

The agents keep their own records

The agent setup itself is documented the same way, because it changes almost every day:

  • A timeline, newest entry first. Facts tied to a date, only ever added to, never edited.
  • A lessons file, each entry an observation and a lesson. It has more than thirty entries now.
  • Decision records: 36 of them between 17 October and 11 November, each with the context, the options considered, the decision and its costs.

The point isn't the paperwork. It's that the next session, mine or an agent's, can read why
something is the way it is, not just what it is.

The day I rebuilt everything

On 31 October I reinstalled the machine the agents run on from scratch. By the end of the next day
everything was back: the agents, the masked test environment the investigator uses, the read-only
data gateway, the monitors from part 2. I brought it back in three stages and checked each stage
before the next.

That was only possible because the setup steps, the rules and the reasons were all written down.
I improved the setup guide along the way, for the next time.

It also showed the limit of documentation. One monitor's history lived only in a local database on
that machine, and it was lost. Documents let you rebuild how something works. They don't bring back
data. Write down how to rebuild; back up what can't be rebuilt.

A habit, not a project

None of this needs a big documentation effort. It needs a habit at the end of each piece of work.
In my guide I suggest closing every session by asking the agent to:

  1. add a few lines to the work log: what was done, what came of it, what's left;
  2. add anything we corrected today that should hold next time to the rules file;
  3. suggest the first request to paste in when the work picks up again.

The third one is the most useful. The next session starts where this one stopped, instead of
rediscovering it.

And a short list of what not to write down: full chat transcripts (nobody rereads them, keep the
conclusions), personal data or secrets, and copies of things that already live somewhere else.
Copies always drift. Link to the original.

What I took from it

  • Records are the agent's memory. Don't assume the next session has this one's context; write it down and check it loads.
  • Give cross-cutting facts their own home. Facts about how systems connect belong to no single repository.
  • Reuse knowledge people already wrote. The traps are the valuable part, and code won't tell you.
  • Make staleness visible. Name each doc's sources, date the last check, and let a script complain.
  • Close the loop back to the docs. Let the agent report what it found wrong, and check that the reports actually arrive.
  • Write down how to rebuild, back up what can't be rebuilt.

The series

  1. From doing the work to delegating it
  2. Let code decide, let the LLM explain
  3. The rules I never break
  4. Context is the job
  5. Agents forget, so I write it down (this post)
  6. kenny: an agent that does my job
  7. Hermes: reminders that carry the check
  8. How to start in two weeks

What's the one question about your system that someone, or something, has to answer by reading the
code from scratch every time?

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.