Dev.to AI 🤖 Ai 👁 0 📖 7 min read

Proactive AI Needs to Remember What You Decided. Give Your LLM Wiki a Reflection Pass.

Two things kept showing up in my feed this week. The first was the "stop building your second brain from an empty folder" posts: Obsidian's CEO publishing his vault template, Karpathy's LLM Wiki pattern, and Claude-insi

Proactive AI Needs to Remember What You Decided. Give Your LLM Wiki a Reflection Pass.

Two things kept showing up in my feed this week.

The first was the "stop building your second brain from an empty folder" posts: Obsidian's CEO publishing his vault template, Karpathy's LLM Wiki pattern, and Claude-inside-Obsidian plugins with tens of thousands of stars. People are not short on capture tools anymore.

The second was First Person Labs launching Second, a personal model that runs on your Mac and hands agents your context over an API. In an a16z clip, CEO Ken Liu splits "proactive" AI in two. Reacting to an email is the easy part. The hard part is what he calls reflective: "Months ago you said this, and..."

That second half is the part almost every LLM wiki I've seen is missing. We have gotten good at helping agents write notes and okay at helping them find notes. We are still bad at having the knowledge base point out, unprompted, that a new fact undercuts something we decided in March.

You don't need a new app watching everything you do to get a first version of that. One reply under the launch said roughly what I was thinking: the context part is what agents are missing, but I'm not sure I want one more app watching everything. So this post builds a small, boring, local version: decision notes with a few frontmatter fields, plus a ~100-line script that produces a reflection digest your agent reads at the start of a session.

The reflection loop: decision note, new notes, reflect.py digest, agent session, then revise or reaffirm

Why retrieval doesn't give you reflection

Retrieval answers the question you asked. Reflection answers a question you forgot to ask.

If I open a session and say "help me tune the sync queue", a decent RAG setup will pull the queue design doc. It probably won't bring up that:

  • we picked SQLite in March on the assumption of fewer than 50 writers per workspace,
  • we promised to revisit that by September 1,
  • and a perf note from October 28 says traces show write contention past ~70 editors.

None of those three facts is hidden. They sit in three files written months apart, and nothing connects them, because connecting them needs a notion of time and commitment that plain Markdown doesn't have.

So add that, in the smallest form that still works.

Step 1: Make decisions a note type

A decision note is just Markdown with five frontmatter fields:

---
type: decision
decided: 2026-03-04
status: active            # active | superseded
revisit_by: 2026-09-01
watch: [sqlite, write contention, queue]
---
# Use SQLite for the sync queue

Assumption: fewer than 50 writers per workspace.
Revisit if write contention shows up in traces.

Anatomy of a decision note: decided, status, revisit_by, watch terms

The field that does the work is watch. It's the list of words that, if they show up in a newer note, mean past-you would want to hear about it. Write them when you decide, while you still remember what would change your mind. That's a very different question from "what tags describe this?"

revisit_by is the second half: a decision with no expiry date turns into folklore. And status: superseded plus superseded_by: <note-id> keeps the history readable instead of quietly editing the old note.

Regular notes only need a date: field. Todos follow one convention: - [ ] task (owner: name, due: YYYY-MM-DD).

Step 2: The reflection pass

reflect.py reads the vault and never writes to it. It reports four kinds of items:

Kind Meaning
OVERDUE an active decision is past its revisit_by date
NEW-EVIDENCE a note dated after the decision mentions one of its watch terms
BROKEN a superseded decision points at a note that doesn't exist
STALE-TODO an open checkbox is past its due date

Here's the core of it (stdlib only):

def reflect(notes, today, topic=None):
    ids = {n["id"] for n in notes}
    decisions = [n for n in notes if n["meta"].get("type") == "decision"]
    out = []
    for d in decisions:
        m = d["meta"]
        watch = m.get("watch", []) or []
        if topic and topic.lower() not in [w.lower() for w in watch]:
            continue
        if m.get("status") == "superseded":
            sup = m.get("superseded_by", "")
            if sup and sup not in ids:
                out.append(("BROKEN", d, f"superseded_by '{sup}' does not exist"))
            continue
        rv = date(m.get("revisit_by"))
        if rv and rv < today:
            out.append(("OVERDUE", d, f"revisit_by {rv} ({(today - rv).days} days ago)"))
        for n in notes:
            if n is d or not n["date"] or not d["date"] or n["date"] <= d["date"]:
                continue
            if n["meta"].get("type") == "decision":
                continue
            hits, line = first_hit(n["body"], watch)
            if hits:
                out.append(("NEW-EVIDENCE", d, f"{n['path']} ({n['date']}) mentions {', '.join(hits)}: \"{line[:70]}\""))
    for n in notes:
        for task, owner, due in TODO.findall(n["body"]):
            if date(due) < today and (not topic or topic.lower() in task.lower()):
                out.append(("STALE-TODO", n, f"{task} (owner: {owner.strip()}, due {due})"))
    return out

The rest is a 15-line frontmatter parser (a flat YAML subset: key: value and [a, b] lists), a word-boundary matcher, and argparse. It exits non-zero on OVERDUE or BROKEN, so you can run it in CI or a pre-commit hook if your wiki lives in git.

I ran it against a six-file test vault: three decisions, three dated notes.

reflect.py output on the test vault: one overdue decision, two new-evidence hits, one broken supersede link, one stale todo

Five items, and every one is something a person would want flagged before tuning that queue:

  • The SQLite decision is 79 days overdue for review.
  • The October perf note mentions "write contention" and "queue". That's the exact evidence the March decision said would trigger a revisit.
  • The "no vector DB yet" decision gets a hit too, but read the line: recall was 83%, above the 80% threshold. That evidence supports the decision. The script doesn't judge, it surfaces. Reaffirming a decision with a dated link to the eval is just as useful as reversing one.
  • A superseded decision points at a note nobody ever wrote, so the replacement decision only exists in someone's head.
  • The load test that would settle the SQLite question was due November 7 and never happened.

With --topic recall it narrows to just the retrieval decision. That's the mode you want when an agent is about to work in one area.

Step 3: Wire it into the agent session

The digest is short on purpose. Five lines cost far less context than re-reading the vault, and they're the five lines most likely to change what the agent does. In your AGENTS.md (or whatever your harness reads first):

## Before changing anything in an area
1. Run `python3 tools/reflect.py wiki --topic <area>`.
2. If any OVERDUE or NEW-EVIDENCE item touches the task, say so first
   and ask whether to revise or reaffirm the decision.
3. Never edit a decision note's body. To change course, write a new
   decision note and set the old one to status: superseded.
4. To reaffirm, append a dated line linking the evidence note and bump
   revisit_by. Do not delete history.

Rule 3 matters more than it looks. If the agent can rewrite old decisions, you lose the "months ago you said this" property, the whole thing you were building. Superseding keeps the chain intact. The BROKEN check makes sure the chain doesn't quietly end at a missing note.

If you'd rather not hand-roll the loop, this is the kind of structure a good AI knowledge base software setup should give you anyway: typed pages, properties you can query, and backlinks that work on dates as well as titles. We keep ours in a workspace that's an open source Notion alternative and run the script on a Markdown export. The fields map cleanly onto page properties.

What the script does badly (on purpose)

Keyword matching is crude. "Queue" will hit unrelated notes in a big vault. This matters less than it sounds, because watch terms are chosen per decision and checked only against newer notes. If it gets noisy, swap first_hit for an embedding similarity check. Keep the time filter, though. That's what makes it reflection and not search.

It doesn't read meaning. It can't tell confirming evidence from contradicting evidence. That's fine: the step it hands to a human or an agent is the cheap one (read two short lines and judge). The expensive step was noticing those two lines belonged together.

It trusts your dates. No date:, no reflection. That's the one bit of discipline the system asks for, and it's also what makes any knowledge base software useful a year later.

Keep the reflective layer local

The Second launch and the replies under it are a good preview of the next fight. Context is the valuable part, and whoever holds it holds the leverage. A decision log is about the most sensitive context a team has: what you bet on, what you assumed, what you never got around to testing.

That's an argument for keeping this layer on your own machine or your own server, in files you can grep. We wrote more about treating context management as an infrastructure layer and not as an app feature. If you want the model side local as well, the trade-offs are in this guide to self-hosted AI for teams.

And if you're still at the empty-folder stage everyone on X is warning about, start with decisions, not with capture. A second brain that only stores what you read is a library. One that remembers what you committed to, and tells you when the world moved, is closer to the "proactive" thing everyone is pitching.

Try it: drop three of your real decisions into notes with watch and revisit_by, run the pass, and see what it finds. The six-file test vault turned up an overdue decision on the first run. A real one probably will too.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.