OpsRecall: An AI-Assisted Incident Management and Organizational Memory Platform

A production incident is rarely difficult only because something has failed. It is difficult because the engineer investigating it may not know that the same problem happened before, what was tried, which workaround failed, or which resolution was actually verified.

I built OpsRecall to address this gap. It connects incident management with organizational memory, allowing engineering teams to preserve verified resolutions and retrieve relevant experiences during future investigations.

1. Incident tracking is not the same as organizational memory

Software systems experience recurring operational problems: Redis cache failures, API timeouts, database connection issues, deployment errors, authentication failures, and resource exhaustion.

During an incident, engineers inspect logs, investigate possible causes, test solutions, and eventually resolve the issue. However, the knowledge gained during this process is often scattered across tickets, chat conversations, documentation, and individual experience.

A conventional incident management system can record what happened. But recording an incident does not automatically make its resolution available when a similar issue occurs again.

That is the problem I wanted OpsRecall to address.

OpsRecall provides a unified workflow to:

• Create and manage operational incidents.
• Attach technical logs and diagnostic information.
• Investigate incidents with AI assistance.
• Retrieve relevant historical experiences.
• Record verified resolutions.
• Preserve those resolutions in organizational memory.
• Compare investigations with and without historical memory.

The main idea is straightforward: an incident should not end with a status change. It should leave behind useful engineering knowledge.

2. How I structured the system

I separated OpsRecall into four main technical responsibilities.

Frontend: React, TypeScript, Vite, and Tailwind CSS provide the application interface. The frontend includes the overview dashboard, incident management, investigation workspace, memory explorer, and evaluation views.

Backend: Python and Flask provide the REST API. The backend handles request validation, incident workflows, database operations, and Hindsight integration.

Operational database: PostgreSQL stores incidents, technical logs, resolutions, investigation runs, and memory synchronization status.

Organizational memory: Hindsight Cloud provides the memory operations used to retain, retrieve, and reason over previous incident experiences.

The browser communicates with Flask through REST APIs. Flask coordinates PostgreSQL and Hindsight. Database credentials and Hindsight API credentials remain on the backend rather than being exposed to the browser.

I deliberately keep PostgreSQL and Hindsight separate. PostgreSQL is the system of record for operational data. Hindsight provides the memory layer that makes previous experiences useful in later investigations.

3. The important design decision: commit the resolution before retaining memory

One of the most important decisions in the implementation was deciding what should happen when an incident is resolved.

Hindsight is an external service. A network failure, timeout, or provider-side issue should not undo a resolution that has already been confirmed and saved.

For that reason, OpsRecall first commits the resolution and updates the incident status in PostgreSQL. Only after the database transaction completes does it prepare the experience and call Hindsight Retain.

The simplified flow looks like this:

resolution = commit_resolution_and_close_incident(payload)

# Database transaction is complete before external retention.
experience = format_incident_experience(resolution)
safe_experience = scrub_credentials(experience)

document_id = f"opsrecall-incident-{resolution.incident_id}"

try:
    retain_in_hindsight(
        bank_id=PRIMARY_MEMORY_BANK,
        document_id=document_id,
        content=safe_experience,
    )
    mark_memory_synced(resolution.id)

except HindsightError as error:
    mark_memory_failed(
        resolution.id,
        sanitize_error(error)
    )

The resolution record tracks four memory-related fields:

• memory_status
• memory_document_id
• memory_synced_at
• memory_error

The synchronization status can be pending, synced, or failed. A retry endpoint allows the system to attempt retention again without requiring the engineer to recreate the resolution.

I also use a stable document identifier derived from the incident UUID. This means a retry refers to the same logical incident experience rather than creating a new document every time.

4. Memory quality starts before Hindsight

I did not want to retain every line of an incident conversation as if it were a verified solution.

A useful operational memory needs context and an outcome. OpsRecall formats a resolution into a structured experience containing the incident context, symptoms, root cause, corrective action, and outcome.

I distinguish three outcome categories:

OUTCOME_LABELS = {
    "successful": "VERIFIED SUCCESSFUL RESOLUTION",
    "partial": "PARTIAL RESOLUTION (INCOMPLETE MITIGATION)",
    "failed": "FAILED TROUBLESHOOTING ATTEMPT",
}

This distinction matters.

A failed troubleshooting attempt can still be useful to a future engineer, but it should not be presented as a proven fix. A partial mitigation should not be confused with a complete resolution.

Before retention, OpsRecall also sanitizes sensitive information such as password assignments, bearer tokens, API keys, and private-key material. If unredacted credentials remain, the retention process blocks the payload.

I consider this a core part of memory design. Information stored today may be retrieved in a future investigation, so sensitive values must be handled before they cross the memory boundary.

5. Why I integrated Hindsight

I use Hindsight, the open-source agent memory framework, as the organizational memory layer rather than trying to implement all memory retrieval and contextual reasoning inside the incident database.

The integration is based on three operations:

Retain stores a structured incident experience after a resolution is recorded and sanitized.

Recall retrieves relevant historical experiences when a new incident is investigated.

Reflect considers the current incident together with retrieved context to support a structured analysis.

The Hindsight documentation provides further details about these memory operations. The broader concept of agent memory and persistent context is particularly relevant here: an AI system can make use of information from previous interactions instead of treating every request as an isolated task.

For a memory-enabled investigation, Flask loads the incident and its technical logs, requests relevant experiences from Hindsight Recall, and passes the current incident context and retrieved evidence into Reflect.

The resulting investigation is stored in PostgreSQL, including its memory mode, evidence, recommendations, and response-time information.

I also maintain two separate memory banks:

• opsrecall-demo is the primary bank used for Retain and memory-enabled investigations.
• opsrecall-baseline is the baseline bank used for investigations without organizational memory.

Bank selection is handled on the backend:

def bank_for_mode(memory_mode: str) -> str:
    if memory_mode == "enabled":
        return PRIMARY_MEMORY_BANK

    if memory_mode == "disabled":
        return BASELINE_MEMORY_BANK

    raise ValueError("Unsupported memory mode")

Keeping the baseline bank separate prevents the memory-disabled investigation path from accidentally using the primary organizational experiences.

6. Comparing memory-enabled and baseline investigations

I wanted to make the contribution of historical memory inspectable rather than simply claiming that memory improves every answer.

OpsRecall supports two investigation modes:

• Memory-enabled: The investigation can retrieve relevant experiences from the primary Hindsight bank.
• Baseline: The investigation runs without access to the primary organizational memory.

The comparison workflow runs both modes against the same incident context and associates the resulting records with a shared comparison identifier.

A simplified example:

comparison_id = str(uuid.uuid4())

memory_run = investigate(
    incident=incident,
    memory_mode="enabled",
    comparison_id=comparison_id,
)

baseline_run = investigate(
    incident=incident,
    memory_mode="disabled",
    comparison_id=comparison_id,
)

Each investigation keeps its own identifier, memory mode, evidence, and response time. The shared comparison ID connects the two results.

This allows engineers to review which historical experiences were retrieved, whether the evidence supports a suggested root cause, and how recommendations differ between the two modes.

I do not treat this as a scientific benchmark. Model behavior, network conditions, and retrieval results can vary. The comparison interface is intended to support inspection and evaluation, not to prove that memory-enabled analysis is universally better.

7. The interface is part of the trust model

An AI-generated answer without evidence is difficult to trust during an operational incident. Engineers need to inspect the current incident, retrieved memories, recommendations, and eventual resolution.

I built the frontend around that workflow.

The main application areas include:

Overview Dashboard: Summarizes incident status, severity, affected services, and system health.

Incident Management: Supports incident creation, searching, filtering, pagination, technical logs, status changes, and resolution capture.

Investigation Workspace: Provides memory-enabled investigation, baseline investigation, and side-by-side comparison.

Organizational Memory Explorer: Displays retained experiences, memory synchronization status, and natural-language search against Hindsight.

Evaluation Workspace: Makes paired investigation results reviewable through evidence, recommendations, retrieved-memory counts, response timing, and a human evaluation rubric.

The Hindsight integration also required careful asynchronous runtime handling. Flask request processing and an asynchronous HTTP client do not automatically share a safe event-loop lifecycle. I addressed this by using a managed Hindsight runtime with a dedicated event loop and thread, allowing Flask requests to submit asynchronous operations safely.

8. What I learned while building OpsRecall

A few engineering lessons stood out during implementation.

1. External services should not own operational transactions.

A verified resolution must remain saved even when memory synchronization fails. Commit the source-of-truth record first, then handle external retention with explicit status and retry behavior.

2. Memory needs provenance and outcome information.

Storing a large block of text is not enough. A future investigation needs to know what happened, what was attempted, whether it worked, and which incident the experience belongs to.

3. Security must be part of the retention workflow.

Redaction should happen before information enters long-term memory. If credentials remain, the system should reject the payload rather than preserve sensitive information.

4. Compare equivalent investigations.

A meaningful memory-enabled versus baseline comparison needs the same incident context and a reliable identifier connecting both runs.

5. Test the runtime, not just individual functions.

Unit tests are important, but asynchronous client lifecycle, event-loop ownership, cleanup, and external network behavior must also be tested through the actual application workflow.

Conclusion: the incident should teach the system something

OpsRecall is built around a simple operational principle: closing an incident should preserve more than a status change.

A verified resolution should become a structured, sanitized, outcome-labelled experience that can help an engineer investigate a related failure in the future.

Historical memory is not infallible, and it does not replace logs, metrics, tracing, or engineering judgement. It adds another source of context: what the organization has already learned, what it tried, and what actually worked.

The goal is not simply to generate an AI response. It is to make today's verified fix available as tomorrow's investigation context.

Implementation stack: React · TypeScript · Vite · Tailwind CSS · Python · Flask · PostgreSQL · SQLAlchemy · Hindsight Cloud