Agent Memory Architecture – A Practical Guide

📋 Concise Diagnosis Table

⚠️ Editorial Caveats

• A symptom does not prove a specific root cause; systematic investigation (labels, context, preprocessing, base‑model limits) is required before attributing errors to pre‑training or fine‑tuning.
• Retrieval‑Augmented Generation (RAG) improves information access but does not guarantee factual accuracy, instant compliance, or complete coverage.
• Low‑Rank Adaptation (LoRA) freezes base weights and learns low‑rank updates; it does not guarantee preservation of all existing behavior. Regression testing outside the target task is essential.
• Document retrieval and live‑value API calls are distinct; cached documents must never replace a required live‑value fetch.

🧭 Decision Flow (Mermaid)

flowchart TD
    A["Incoming request"]
    B["Is request for volatile data?"]
    C["Use Working Memory"]
    D["Is request for user preferences?"]
    E["Use Durable Preferences"]
    F["Is request for audit trace?"]
    G["Use Episodic Memory"]
    H["Is request for verified fact?"]
    I["Use Authoritative Facts"]
    A --> B
    B -->|yes| C
    B -->|no| D
    D -->|yes| E
    D -->|no| F
    F -->|yes| G
    F -->|no| H
    H -->|yes| I
    H -->|no| C

🏗️ Layer‑by‑Layer Responsibilities

Key principles

1. TTL enforcement – each layer has a hard expiration enforced by the runtime, never by “best‑effort”.
2. Deletion & privacy – Durable Preferences and Episodic records must support explicit user‑initiated deletion workflows that are auditable.
3. Staleness guardrails – Authoritative Facts must never be served beyond their freshness window; a deterministic check (now - cached_at < MAX_FRESHNESS) decides whether to use the cache or hit the live source.

🛠️ Python Decision Helper (Teaching‑Only)

# -*- coding: utf-8 -*-
"""Memory layer dispatcher – teaching only.

The helper decides which memory store a request should hit based on
metadata that the orchestration layer supplies.  It contains only
standard‑library constructs and heavy assertions to illustrate the
contract between the caller and each memory layer.
"""

import datetime as _dt
from enum import Enum, auto
from typing import NamedTuple, Literal

# ----------------------------------------------------------------------
class MemoryLayer(Enum):
    """Enumerates the four memory layers."""
    WORKING = auto()
    DURABLE = auto()
    EPISODIC = auto()
    AUTHORITATIVE = auto()

class Request(NamedTuple):
    """A minimal request description used by the dispatcher."""
    kind: Literal["volatile", "preference", "audit", "fact"]
    created_at: _dt.datetime  # UTC timestamp of request arrival
    ttl_seconds: int | None = None  # Optional explicit TTL

# ----------------------------------------------------------------------
def select_layer(req: Request) -> MemoryLayer:
    """Return the appropriate MemoryLayer for *req*.

    Assertions act as guard rails – they fail fast if the caller supplies
    contradictory information, which mirrors the deterministic business
    rules required in production.
    """
    now = _dt.datetime.utcnow()
    age = (now - req.created_at).total_seconds()

    # 1️⃣ Volatile data → Working Memory (must respect TTL)
    if req.kind == "volatile":
        assert req.ttl_seconds is not None, "TTL required for volatile data"
        assert age <= req.ttl_seconds, "Request exceeds its TTL"
        return MemoryLayer.WORKING

    # 2️⃣ User preference → Durable Preferences (no TTL, versioned)
    if req.kind == "preference":
        assert req.ttl_seconds is None, "TTL must not be set for preferences"
        return MemoryLayer.DURABLE

    # 3️⃣ Audit trail entry → Episodic Memory (append‑only)
    if req.kind == "audit":
        assert req.ttl_seconds is None, "TTL not applicable to audit logs"
        return MemoryLayer.EPISODIC

    # 4️⃣ Verified fact → Authoritative Facts (freshness check)
    if req.kind == "fact":
        # Example: allow cache only if less than 5 min old
        MAX_FRESHNESS = 300
        assert age <= MAX_FRESHNESS, "Cached fact is stale – fetch live"
        return MemoryLayer.AUTHORITATIVE

    raise ValueError(f"Unsupported request kind: {req.kind}")

# ----------------------------------------------------------------------
# Example (not executed in production):
if __name__ == "__main__":
    r = Request(kind="volatile", created_at=_dt.datetime.utcnow(), ttl_seconds=30)
    assert select_layer(r) == MemoryLayer.WORKING

📜 Illustrative Architecture Decision Records

ADR 001 – Separate Working Memory from Durable Preferences

Context

Early prototypes stored both session‑level variables and user‑level settings in a single key‑value map. Privacy audits revealed that “delete my data” requests inadvertently left preference remnants, and latency spikes occurred because volatile data crowded the persistent store.

Decision

Introduce a distinct Working Memory store scoped to a session, backed by an in‑process LRU with a hard TTL (SESSION_TTL). Move all user‑level settings into Durable Preferences, persisted in a versioned KV table with an explicit deletion API.

Consequences

Acceptance Criteria

• All “delete my data” automated tests verify that after the API call, no keys remain in the Durable store for the user.
• End‑to‑end latency for a 10‑turn conversation drops ≤ 15 % in load tests.

Uncertainty – The exact TTL value may need tuning based on observed session lengths; monitoring will guide adjustments.

ADR 002 – Cache Authoritative Facts Only When Freshness Is Proven

Context

The bot frequently queried a live tax‑rate service. To improve responsiveness, a naive cache without expiry was introduced, leading to regulatory violations when rates changed.

Decision

Implement a Authoritative Facts cache that stores the raw response together with cached_at. Every read must run a deterministic freshness predicate (now - cached_at < MAX_FRESHNESS). If the predicate fails, the system must invoke the live API; fallback to cache is prohibited.

Consequences

Acceptance Criteria

• In integration tests, any simulated rate change after MAX_FRESHNESS must result in a different value being returned.
• Production monitoring shows > 95 % cache‑hit rate while never serving a fact older than MAX_FRESHNESS.

Uncertainty – External service latency variance may affect overall response time; adaptive back‑off strategies will be evaluated if miss latency exceeds a defined threshold.

✅ Evaluation & Maintenance Checklist

• [ ] TTL Enforcement – Unit tests verify that select_layer raises on expired volatile requests.
• [ ] Privacy Deletion – Automated job runs nightly to confirm no residual keys for users who invoked the deletion endpoint.
• [ ] Episodic Pruning – Scheduled job trims records older than MAX_EPISODE_AGE; post‑run count must be ≤ target retention size.
• [ ] Authoritative Freshness – Integration test injects a stale cache entry; request must trigger a live API call (mocked).
• [ ] Performance Regression – Benchmark end‑to‑end latency before and after layer separation; reject changes that increase 95th‑percentile latency by > 10 %.
• [ ] Regression Suite – Include prompts that require context retrieval, live‑value calls, and deterministic business‑rule evaluation; verify no drift after any code change.
• [ ] Documentation Sync – Ensure the mermaid diagram, table definitions, and ADRs are version‑controlled alongside code.

📚 References

1. Microsoft Retrieval‑Augmented Generation (RAG) concepts – https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
2. LoRA: Low‑Rank Adaptation of Large Language Models – https://arxiv.org/abs/2106.09685