Why My Hindsight Agent Stores What Failed,Not Just What Worked

The most useful thing my on-call agent has ever told me was three words long: don't restart it.

It was 2 a.m. in a staging replay of a payments incident, and every instinct I have said "restart the pods." The agent pulled up an incident from months earlier where someone had done exactly that, the connection pool had gone from degraded to dead, and the postmortem had recorded the outcome as WORSENED. It cited the incident ID, explained why the restart backfired, and suggested what had worked in a different incident instead.

That behavior doesn't come from a clever prompt. It comes from one decision about what goes into memory: I store failures with the same weight as fixes.

What the system does

The Incident Response Agent briefs whoever is on call. When an alert fires, it looks up what the company has seen before, and produces a short briefing: what this looks like, which past incidents are relevant, what was tried, and what happened when it was tried.

The pieces:

• Spring Boot service that receives alerts and owns the briefing flow.
• PostgreSQL as the system of record for incidents, services, alerts, and actions taken. If memory disappeared tomorrow, I could rebuild it from here.
• Hindsight, the open-source agent memory system, as the long-term memory the agent recalls from.
• Gemini for turning recalled memories into a readable briefing.
• A side panel in the UI that shows exactly which memories were recalled for the current alert, next to the briefing.

The target environment is a company with a handful of microservices (payments, checkout, inventory sync, notifications) and a few years of incident history. The agent's whole job is to make that history available at the moment it matters.

The through-line: an incident is a sequence of actions, and actions have outcomes

Most "incident memory" designs I've seen store the postmortem summary: what broke, root cause, fix. That's a record of the ending. It's the least useful part when you're mid-incident, because mid-incident you don't know the root cause yet. You know the symptoms and you're deciding what to try next.

The question an on-call engineer actually asks is: the last time we saw this, what did people try, and what did it do? Some of those actions fixed things. Some did nothing. Some made it worse. The last category is the one that's lost when people write "resolved by increasing pool size" in a postmortem and forget that the first two attempts made it worse.

So I model memory at the action level, and every action carries an outcome:

public enum ActionOutcome {
    RESOLVED,    // incident cleared after this action
    MITIGATED,   // symptoms reduced, root cause still present
    NO_EFFECT,   // nothing observable changed
    WORSENED     // metrics or user impact got worse after this action
}

public record IncidentAction(
        String incidentId,        // e.g. "INC-2041"
        String service,           // e.g. "payments-api"
        String symptom,           // what the alert looked like
        String action,            // what the engineer did
        ActionOutcome outcome,
        String causalExplanation  // why it had that effect
) {}

The causalExplanation field is the one I fought for. A bare WORSENED tag tells you not to do something. A causal explanation tells you whether the warning applies to the incident in front of you. "Restart made it worse" is a superstition. "Restart made it worse because all pods reconnected to Postgres at once and exhausted max_connections" is something I can compare against the current situation and decide whether it applies.

How Hindsight fits

I didn't want to build retrieval myself. An action record like the one above is a few sentences of natural language plus some structure, and what I need back is "the records most relevant to this symptom on this service," not an exact match. Hindsight's agent memory gives me retain and recall operations that handle that: I write memories in, and I query them back by meaning, with the metadata I attach to each one.

Writing a memory is one call. I render each action into a self-contained sentence, because a memory that depends on surrounding context gets misread when it's recalled alone:

public void retain(IncidentAction a) {
    String content = """
        Incident %s on %s. Symptom: %s. Action taken: %s. \
        Outcome: %s. Why: %s""".formatted(
            a.incidentId(), a.service(), a.symptom(),
            a.action(), a.outcome(), a.causalExplanation());

    hindsight.retain(
        bankId,
        content,
        Map.of("incident_id", a.incidentId(),
               "service", a.service(),
               "outcome", a.outcome().name()));
}

The outcome sits in the text, where semantic recall can see it, and also in metadata, where my code can check it. I use both on purpose. The text lets a query like "payments connection errors after deploy" pull back the right memories. The metadata lets the application, not the model, decide which recalled memories count as warnings.

If you're new to the idea, Vectorize has a good overview of what agent memory is and how it differs from stuffing history into a prompt. The short version of why I wanted it here: an incident history grows forever, and I don't want to re-summarize it on every alert.

The rule I'm strictest about: recall and reasoning stay separate

Early on I noticed the failure I was most afraid of. If I hand an LLM a pile of context and ask for a briefing, it will produce a fluent one whether or not the context supports it. A fabricated warning ("a similar incident in March was made worse by a failover") is worse than no warning, because it sounds like institutional knowledge.

So the pipeline has two stages with a hard wall between them:

1. Recall. Query Hindsight for memories relevant to the alert. No LLM writes anything in this step.
2. Briefing. Pass only the recalled memories to Gemini, with each one labeled by incident ID, and instruct it to cite those IDs.

Then I check the output in code instead of trusting it:

public Briefing build(Alert alert) {
    List<RecalledMemory> recalled = hindsight.recall(bankId, alert.asQuery(), 10);
    Set<String> knownIds = recalled.stream()
            .map(m -> m.metadata().get("incident_id"))
            .collect(Collectors.toSet());

    Briefing draft = gemini.brief(alert, recalled);

    for (Warning w : draft.warnings()) {
        boolean traceable = knownIds.contains(w.incidentId());
        boolean backedByFailure = recalled.stream().anyMatch(m ->
            m.metadata().get("incident_id").equals(w.incidentId())
            && "WORSENED".equals(m.metadata().get("outcome")));
        if (!traceable || !backedByFailure) {
            throw new UngroundedWarningException(w);
        }
    }
    return draft;
}

A warning is only allowed through if it cites an incident that was actually recalled and that incident's recorded outcome was WORSENED. If Gemini invents an ID, or cites a real incident that doesn't support the claim, the briefing is rejected and regenerated. This was the most annoying code to write and the part I'd refuse to ship without. The model is good at writing the briefing; it isn't the authority on whether the briefing is true.

The side panel is the same principle shown to a human. Whatever Hindsight returned is displayed verbatim next to the briefing, so an engineer who doubts a warning can read the source memory in two seconds.

What it looks like in practice

Take two incidents. The first, INC-2041, is the one with no useful history: payments-api starts throwing connection timeouts, and the agent has nothing similar to recall. It says so. The briefing is generic and explicitly marked as having no prior incidents, which matters more than it sounds, because an agent that always produces confident history teaches people to ignore it.

During INC-2041 the on-call engineer restarts the pods. Connection errors get worse. That sequence gets retained as an action record with outcome WORSENED and a causal explanation: every pod reconnected simultaneously, saturating Postgres max_connections. Later in the same incident, raising the pool limit on a single instance and draining traffic gradually is recorded as RESOLVED.

Months later, checkout-service alerts on rising latency with intermittent database connection errors. Different service, different symptom wording, same underlying shape. The briefing reads roughly like this:

Similar to INC-2041 (payments-api, connection exhaustion).

Warning: In INC-2041, restarting all pods at once was recorded as WORSENED.
Cause on record: simultaneous reconnects saturated Postgres max_connections.
Check current connection count before restarting anything.

What worked in INC-2041 (RESOLVED): raising the pool limit on one instance
and draining traffic gradually.

No other relevant incidents recalled.

Every line there traces to a recalled memory and an incident ID. The agent isn't saying the two incidents are identical. It's saying one failure mode might apply and showing the evidence.

Lessons learned

Record what was tried, not just what worked. The most valuable memories were the ones that stopped someone from repeating a mistake. If your agent only remembers successes, it has no way to know that the obvious fix is a trap.

Store the "why" with every outcome. An outcome label without a causal explanation is a rule you can't check against a new situation. The explanation is what lets the agent, and the human reading it, decide whether a past failure is relevant.

Put structure in the metadata and meaning in the text. Natural-language memories are great for recall by meaning. But the decisions your application makes, like "is this a warning?", should read structured fields, not parse prose.

Don't let the model grade its own output. Retrieval and generation are separate steps for a reason. I validate every warning against what was recalled, and rejecting ungrounded output in code is cheaper than discovering it during an outage.

Make "I have no history for this" a first-class answer. Letting the agent say it has nothing relevant is what makes its warnings believable when it does have something. It took more design effort than any retrieval tuning.

If you want to try this pattern, the Hindsight documentation covers the retain and recall operations I used, and the code is open source on Hindsight's GitHub repository. My version is one specific use of it, but the core idea transfers to anything where the cost of repeating a past mistake is high: remember the failures.