Self-healing browser actions for AI agents (and why they must refuse to guess)

When an LLM drives a browser, the classic failure isn't "the model is dumb". It's that the page
moved under its feet. The agent reads the page, decides to click e3, and by the time the click
lands, a framework re-render has replaced that DOM node with an identical-looking new one. The id is
gone, the click fails, and the model burns tokens re-reading the page and re-deciding.

ecobrowser handles this in code, not in the prompt. The principle is: do work in code so the model
doesn't spend tokens and reasoning on it. Here's how the self-healing path works, and the one rule that
makes it safe.

1. Give elements durable ids

Perception runs a script inside every frame and open shadow root, collects interactive elements, and
stamps each with a durable data-ai-id. Ids are never reused, even across pages, so "e3" means the
same thing five steps later. The agent acts by id, never by guessed CSS selector or pixel
coordinate.

2. Try the action, then heal only if the id vanished

// reliableOnElement (simplified)
const first = await reliable(`${verb} ${id}`, () => act(sel(id)), { retries: 1 });
if (first.ok || first.maybeApplied || cancelRequested) return first;

const newId = await healId(id);
if (!newId) return first;               // refuse to guess
return reliable(`${verb} ${id} (re-located as ${newId})`, () => act(sel(newId)));

Three details matter:

• One cheap retry first. Most flakiness is timing, not a moved id.
• maybeApplied is never retried. If a click's handler hung after the event was dispatched, the action may already have happened. Retrying could submit a form twice. The result is reported honestly as maybeApplied and the agent decides.
• Healing is a second phase, not the default. It only runs when the first attempt failed outright.

3. Re-locate by identity, and refuse when it's ambiguous

healId re-snapshots the page and looks for an element with the same identity: tag, role and
accessible name. It returns null, meaning "I won't guess", in four cases:

1. The id still exists. Then the failure is real (covered, disabled, off-screen), not a moved id.
2. The page is a different document. After a navigation there is nothing to re-locate.
3. The identity was ambiguous before. If the previous snapshot had two "Delete" buttons, the match is meaningless.
4. The identity is ambiguous after. If the new snapshot has zero or several matches, same problem.

Only when exactly one element matches before and after does it hand back the new id.

const same = (e) => e.tag === prev.tag && e.role === prev.role && e.name === prev.name;
if (prevSnap.elements.filter(same).length !== 1) return null;   // ambiguous before
const matches = fresh.elements.filter(same);
return matches.length === 1 ? matches[0].id : null;             // ambiguous after

The comment in the source says it best: acting on the wrong one is worse than failing. A failed action
the agent can see and reason about is cheap. A confident click on the wrong "Delete" button is not.

4. Return a verdict, not a hope

Every action returns an ActionResult: did it work, and did the page change. That distinguishes
success, silent no-op (the click landed, nothing happened) and failure. A MutationObserver
tracks a DOM version, and a fingerprint of each interactive element's label, value and state means a
toggled checkbox or opened menu counts as a change even when no element was added or removed.

What I'd tell someone building a similar layer

• Make "the id moved" a recoverable condition, but make recovery conservative.
• Treat ambiguity as a hard stop. Self-healing that guesses is a bug generator.
• Separate "failed before dispatch" (safe to retry) from "may have applied" (never retry).
• Report outcomes as data the model can use, not just exceptions.

Honest limits

Identity is tag + role + name, so two genuinely different elements with the same label are never
healed (by design), and a renamed element won't be re-found. It's a young, single-user developer tool,
not yet battle-tested across thousands of real sites.

Code: https://github.com/HaidarESBER/ai-browser