An agent can suggest a lesson. Should it become a rule?

If you are running an agentic workflow for coding, I am pretty sure you already know that - sometimes an AI coding agent can get stuck for a stupid, boring reason.

For example, in a recent run of my own workflow, an agent failed because it was missing Jam credentials. I use Jam to report bugs and provide a link and the MCP to the triage agent to help it understand the bug, but this run failed even though there was no Jam link to investigate. If anyone with a common sense would have worked on this issue (i.e. a human) they wouldn't worry about it, but the agent does unless you tell it: Before worrying about access, you needed to check if that access is even relevant.

So in this specific instance the agent stopped, raised a huma-gate, and waited for direction. Once I responded, I would expect the agent to remember that decision... That is the sort of correction I want the next agent to use. But turning every correction into a permanent instruction creates another problem: a growing pile of rules, with nobody checking whether they still belong.

Since I am the one building IssueFlow, this was a practical workflow problem for me. Here is how I approach it.

Start with a specific mistake

“Be more careful” is not much of a lesson. It gives the next agent no useful test.

“Do not block on missing Jam credentials when the issue has no Jam link” is more concrete. It names the condition and the action to avoid.

The important question is whether the proposal captures something useful beyond one task. A workaround for a single issue should not quietly become a rule for every future run.

A proposal is evidence, not an instruction yet

The solution I was after is a balance between being too specific and too broad. I didn't want to send an agent a pile of 200 rules with every prompt. Those will get lost in the sea of instructions, not to mention token cost.

In IssueFlow, agents can submit learning proposals. A librarian agent collects, combines, and generalizes them into readable candidate rules, with the actual agent instructions attached.

The IssueFlow operation (a human, lol) reviews both parts: what the rule means, and what the agent will be told to do. I can approve it, edit it or reject it.

That was an important part I did not want to skip. An agent can draw the wrong lesson from a failure just as easily as it can draw the wrong conclusion while coding. “This failed once” does not automatically mean “never do this.”

A useful rule should explain the mistake it addresses, the condition where it applies, and the behavior it asks for. It should leave enough evidence to check the reasoning.

Choose where the rule belongs

The next part in the learning system design was: which agent should even get those rules. For example, a lesson from 'design-review' does not necessarily belong in 'cleanup'. A rule about implementation does not need to take space in every unrelated stage.
Proposals are being received from a specific agent at a specific stage, and when the librarian works on those learning (generalize, consolidate), it does it for each agent separately.

IssueFlow review process lets me choose specific stages or make a rule general. The stage labels make that scope visible, and I can adjust it.

This is a small decision with a real purpose: keep an instruction close to the work where it matters. More prompt text is not automatically more guidance.

Keep reviewing the rules

Approval is not the end of the process. I wanted to measure the effectiveness of each rule.

Agents report when they use a rule. The librarian can use that history to identify rules not reported as used for a long time, and then suggest retiring the rule. I can also retire a rule manually.

Not only does rule approval matter. Rejection has a purpose too. When I reject a proposal with a reason, that reason is recorded and used by the librarian when it handles similar proposals later. Otherwise, I could spend a whole week just rejecting the same idea in a slightly different hat.

Note: the usage reports are signals to investigate. They are not treated as proof that the agent followed every instruction, or that a particular mistake would certainly have happened without it.

Learning needs human judgment too

What I learned after implementing the self-learning system is that this provides a way to maintain instructions. It is not a promise that agents stop making mistakes, but it does help eliminate many of the recurring problems from real-life experience.

The goal is: make a useful correction available to later work, keep its scope clear, and give people a way to remove advice that no longer helps. Major learning finds its way to the prompt and is retired as a rule.

That leaves a question I’m still interested in: how much instruction should survive from one task to the next?

If you use coding agents, how do you handle that today? Do you keep adding instructions, or do you also review and remove them?

Watch the short IssueFlow workflow demo. It shows a design approval and an unexpected decision during a real run. (The learning process described here is a separate part of the product)