Dev.to AI 🤖 Ai 👁 0 📖 9 min read

Your LLM Wiki Has a Write API Now. Put a Review Gate in Front of It.

This week Obsidian's Steph Ango (kepano) shipped a new batch of Obsidian Skills: instruction packs that teach an AI agent how to work inside a vault, from wikilinks and embeds to .base files, JSON Canvas and the command

Your LLM Wiki Has a Write API Now. Put a Review Gate in Front of It.

This week Obsidian's Steph Ango (kepano) shipped a new batch of Obsidian Skills: instruction packs that teach an AI agent how to work inside a vault, from wikilinks and embeds to .base files, JSON Canvas and the command line. The reaction on X was close to unanimous: "big unlock." Agents stop guessing your formats, and the LLM wiki becomes something an agent maintains instead of something you paste into a chat.

The same weekend, a security researcher posted a short one: a $5k bounty for an auth bypass on an MCP server, where anyone could rewrite a creator's AI knowledge base, no login needed.

Put those two posts side by side and you get the real story of the week. Your second brain now has a write API. Skills make the writes fluent. MCP makes them reachable. Almost nobody has put a review step in front of them.

This post is a practical design for that review step: what an agent should be allowed to write, how to route its edits through a gate, and a small script you can run in CI today. It assumes a Markdown or Git-backed LLM wiki, but the shape works for any knowledge base software that has an API.

Reads leak. Writes rot.

Most of the security talk around AI knowledge bases is about reads: who can see what, and whether the agent leaks a private page into a public answer. That matters. But a bad write is worse in one specific way: it persists and it compounds.

A wrong answer in chat is gone when you close the tab. A wrong sentence written into the wiki gets retrieved tomorrow, summarized into three other pages next week, and cited by a teammate's agent next month. The LLM wiki pattern, where the model compiles raw sources into linked topic pages, is great precisely because the wiki accumulates. Accumulation is also what makes a poisoned or careless write expensive.

There are three ways a bad write gets in:

  1. No auth on the write path. The bounty above. An MCP server exposed a write tool and didn't check who was calling it.
  2. Prompt injection through a source. The agent ingests a web page, a PDF or an email that contains instructions, and dutifully "files" them into a page that every future session will read.
  3. An honest agent being wrong at scale. No attacker at all. The agent merges two pages that sounded similar, rewrites a policy it misread, or "cleans up" a page a human owns.

Auth fixes the first one. Only a review gate handles all three.

Give agents tiers, not a filesystem

The fastest way to build an agent-maintained wiki is to hand the agent a write_file tool scoped to the vault. It's also the riskiest, because every edit has the same power as every other edit.

Split writes into three tiers instead:

Three write tiers for agents: append to inbox, propose a change on a branch, and direct edits reserved for low-risk generated pages

Tier What the agent can do Who approves Good for
Append Create new notes in inbox/ only Nobody, it's a staging area Captures, meeting summaries, "I found this"
Propose Edit anything in wiki/ on its own branch A human merges Topic pages, link maintenance, consolidation
Direct Overwrite a small set of generated files Nobody, but it's reversible Indexes, backlink lists, "recently changed"

Then expose those tiers, not raw file access, as your MCP tools:

[
  { "name": "append_note",
    "description": "Create a new note under inbox/. Cannot modify existing files.",
    "input": { "title": "string", "body": "string", "sources": ["string"] } },
  { "name": "propose_edit",
    "description": "Write a change to wiki/ on branch agent/<run-id>. Returns a review link. Never merges.",
    "input": { "path": "string", "new_content": "string", "reason": "string" } },
  { "name": "rebuild_index",
    "description": "Regenerate wiki/_index.md from current pages. No other effect.",
    "input": {} }
]

Two things fall out of this design for free. First, the server can enforce the tier no matter what the model was told, so a prompt injection that says "overwrite the pricing page" can at most open a proposal. Second, if someone does reach the server without auth, the blast radius is an inbox note or an unmerged branch instead of your whole knowledge base. Fix the auth anyway; AFFiNE's MCP security checklist covers the server side. Tiers are what keep a missed check from becoming a rewrite.

The gate: four checks before a human sees the diff

"A human merges" only works if the human isn't reviewing noise. So put a cheap automated gate in front of the review, and only show people the proposals that pass.

The gate below runs on an agent's branch in a Git-backed wiki. It checks four things:

  • Paths. The agent may touch wiki/ and inbox/ only. No README, no config, no AGENTS.md.
  • Ownership. A page with owner: human in its frontmatter can't be edited by an agent. The agent can propose in inbox/ instead.
  • Provenance. Every wiki/ page needs a sources: list, and every raw/... source it cites must exist in the repo.
  • Instruction smell. Added text that reads like instructions to a model ("ignore previous instructions", a curl command, a base64 blob) is rejected. Knowledge pages describe things; they don't tell the next reader what to do with their tools.

It also caps the size of a single change, because a 900-line "consolidation" is a rewrite, and rewrites deserve a person from the start.

#!/usr/bin/env python3
"""write_gate.py - check an agent's proposed wiki edits before a human merges them.
Usage: python write_gate.py <repo> <base-ref> <agent-branch>
"""
import re, subprocess, sys

ALLOWED = ("wiki/", "inbox/")          # agent may touch these trees only
MAX_LINES = 120                        # big rewrites go to a human first
SUSPICIOUS = re.compile(
    r"ignore (all |any )?(previous|prior) instructions|system prompt|"
    r"curl\s+https?://|base64,|<script", re.I)

def git(repo, *args):
    return subprocess.run(["git", "-C", repo, *args], check=True,
                          capture_output=True, text=True).stdout

def frontmatter(text):
    m = re.match(r"---\n(.*?)\n---\n", text, re.S)
    return m.group(1) if m else ""

def main(repo, base, branch):
    problems = []
    changed = git(repo, "diff", "--numstat", f"{base}...{branch}").splitlines()
    total = 0
    for row in changed:
        added, removed, path = row.split("\t")
        total += int(added) + int(removed)
        if not path.startswith(ALLOWED):
            problems.append(f"{path}: outside allowed paths")
            continue
        try:
            old = git(repo, "show", f"{base}:{path}")
        except subprocess.CalledProcessError:
            old = ""                                   # new file
        if "owner: human" in frontmatter(old):
            problems.append(f"{path}: human-owned page, agent may only propose in inbox/")
        new = git(repo, "show", f"{branch}:{path}")
        fm = frontmatter(new)
        if path.startswith("wiki/") and "sources:" not in fm:
            problems.append(f"{path}: no sources: in frontmatter")
        for src in re.findall(r"^\s*-\s*(raw/\S+)", fm, re.M):
            try:
                git(repo, "cat-file", "-e", f"{branch}:{src}")
            except subprocess.CalledProcessError:
                problems.append(f"{path}: cites missing source {src}")
        diff = git(repo, "diff", f"{base}...{branch}", "--", path)
        added_text = "\n".join(l[1:] for l in diff.splitlines()
                               if l.startswith("+") and not l.startswith("+++"))
        if SUSPICIOUS.search(added_text):
            problems.append(f"{path}: added text looks like instructions, not knowledge")
    if total > MAX_LINES:
        problems.append(f"{total} lines changed (limit {MAX_LINES}); split the change")
    for p in problems:
        print("REJECT", p)
    if not problems:
        print(f"OK {len(changed)} files, {total} lines: ready for human review")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:4]))

I ran it against a tiny test vault with two agent branches. The good branch adds one step to an onboarding page that already cites a meeting note. The bad branch does what a confused or injected agent might: edits a human-owned pricing page and slips in an instruction, adds a page that cites a source that doesn't exist, and touches the README.

Terminal output of write_gate.py: the good branch passes with one line changed, the bad branch is rejected with five reasons

$ python3 write_gate.py kb main agent/good
OK 1 files, 1 lines: ready for human review

$ python3 write_gate.py kb main agent/bad
REJECT README.md: outside allowed paths
REJECT wiki/new.md: cites missing source raw/missing.md
REJECT wiki/pricing.md: human-owned page, agent may only propose in inbox/
REJECT wiki/pricing.md: no sources: in frontmatter
REJECT wiki/pricing.md: added text looks like instructions, not knowledge

To be clear about the limits: the instruction check is a tripwire, not a prompt-injection defense. A careful attacker will phrase around a regex. Its job is to catch the lazy cases and the honest accidents cheaply, so the human reviewer spends attention on meaning. The ownership and path checks are the ones doing real work, because they don't depend on what the text says.

Wire it into the loop

The full write path ends up looking like this:

Agent writes to its own branch, the gate checks paths, ownership, sources and instruction smell, a human reviews the diff and merges, and rejected changes go back to the agent with reasons

  1. The agent calls propose_edit. The server commits to agent/<run-id>, never to main.
  2. CI runs write_gate.py. A rejection goes back to the agent as a tool result, with the reasons, so it can fix provenance or move the change to inbox/.
  3. A passing branch becomes a review item. The reviewer sees a diff plus the reason the agent gave, not a blob of rewritten text.
  4. Merge, or close. Either way the history keeps both the proposal and the decision.

A few habits make this sustainable rather than a new chore:

  • Batch reviews. Merge agent proposals once or twice a day, the way you'd triage an inbox. Most will be small link and wording fixes that take seconds.
  • Let trust expand by path, not by mood. If six weeks of proposals to wiki/glossary/ were all merged unchanged, move that folder to the direct tier. Keep policies, pricing, people and security pages in propose forever.
  • Mark ownership as you go. You don't need to label the whole vault on day one. Add owner: human the first time an agent edit to a page makes you wince.
  • Log the caller. Every write tool call should record which client, which session and which source document triggered it. When something odd lands in the wiki, "which ingest put this here?" should be a lookup, not an investigation.

What to look for in the tool itself

If you're choosing where your team's LLM wiki or second brain lives, rather than building it on a folder of Markdown, the write path is a good filter. Ask:

  • Can I give an agent access to part of the workspace, not all of it?
  • Is there page-level history I can diff and roll back, so an agent edit is reversible?
  • Can I tell an agent edit from a human edit after the fact?
  • Can I host it myself, so the server an agent writes to is one I control and can put behind my own auth?

That last one is why we keep coming back to self-hosted AI for team knowledge. It's also why we build AFFiNE as an open source Notion alternative: when agents are going to write into your AI knowledge base software, you should be able to read the code behind the write path and run it on your own server. Whatever you use, apply the same test to any knowledge base software you're evaluating. Skills and MCP make agent writes easy. Make sure something in your stack makes them reviewable.

The short version

  • Agents can now write to your wiki fluently (skills) and remotely (MCP). Treat that as a write API.
  • Bad writes persist and compound, so they cost more than bad reads.
  • Expose tiered tools (append, propose, direct) instead of raw file access, and enforce the tier on the server.
  • Put a cheap gate in front of human review: paths, ownership, provenance, instruction smell, change size.
  • Widen agent permissions folder by folder, based on what you actually merged.

If you've built a review step for agent writes in your own vault or team wiki, I'd like to hear how you scoped it, especially what you let agents edit directly.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.