The screen is not the state: verifying that a browser action actually happened, and recovering when it did not

A click landed. The button went grey. The log says the action succeeded. Twenty minutes later the record shows nothing happened — and the retry I sent to fix it is the one that actually went through twice.

I build an assistant that operates a browser on someone's behalf: filling forms, submitting paperwork, checking out, cancelling things. The failures that cost me the most were not model failures. They were verification failures — cases where the agent could not tell the difference between "my input reached the page" and "the effect exists in the world", and so it either declared success that never happened or retried something that had already worked.

The class of bug is old. What is new is that an LLM now decides, from a text view of a page, whether its own action succeeded. That decision is the whole product. Here is the model I ended up with, and the specific ways the screen lies.

Three facts, not one

Every web action produces three separate facts, and they are almost never established by the same observation:

1. The input reached the element. The keystrokes or click arrived at the intended control — not an overlay, not a different element with the same label, not a field that re-rendered and dropped it.
2. The page changed in response. Something in the DOM moved: a URL, a status line, a disabled button, a row in a list.
3. The effect persisted somewhere the page does not control. A confirmation number, a record in an account, a message in a recipient's inbox, a charge in a ledger.

Fact 1 is the easiest and least interesting. Fact 2 is what most automation treats as success. Fact 3 is the only one that survives closing the tab — and it is usually checked last, or not at all.

The prompt-level version of this is a single instruction I now put in every browser-capable agent's tool contract: act, then read what changed, then read the record. If you cannot read the record, say so instead of inferring it.

Where the text view of a page lies

An accessibility snapshot — the flat list of interactive elements and text lines an agent reads — is a compressed, lossy rendering of a live document. It fails in specific, predictable ways.

Optimistic UI. A framework updates state before the server answers. The list already shows the new row; the request is still in flight and will fail. Reading the DOM right after the click tells you what the app hoped would happen.

A disabled control is not a finished one. "Print" greys out while the document is being prepared and stays grey for a few seconds. An agent that treats disabled as "already handled" clicks again, or gives up and reports a failure the site never produced. The same shape appears at checkout: a pay button that re-enables after a declined card looks identical to one waiting for a redirect.

Element handles expire. Between the click and the follow-up read, the page re-rendered and every reference the agent held now points at nothing, or at a different node. The symptom is a field that "rejects" input that was typed correctly, or a click that "succeeds" on the wrong row. Recovery is not a retry with the old references; it is a fresh read of the page before touching it again.

Same name, different item. Two "Cancel" buttons, one per booking; a "Continue" that belongs to a different flow. The name matches, the click lands, and the wrong thing happens. The only defence is to read each control in relation to the surrounding record — whose name, which date, which amount — before acting. A description in the plan ("click cancel") is not evidence about which cancel.

The overlay you cannot see in text. A cookie banner, a "what's new" dialog, or a modal prompt covers the target. The click is dispatched, intercepted by the overlay, and reported as a miss — or worse, as a hit on the overlay's own button. If the tool reports that an element was intercepted by another element, that is the page telling you the truth: dismiss the blocker, then re-read.

The masked screen problem

When the agent fills a password, a card number, or a one-time code, a well-built harness refuses to screenshot the page. Good. The trap is what happens next: the agent cannot see pixels, so it looks for another way to convince itself the input took — and the worst version of that is submitting the form to make the screen capturable again.

That is letting a security control change a business decision. The capture restriction is not evidence of failure, and it is not permission to advance the flow. The correct move is to verify through signals that are not secret:

• the field's own state after entry (masked length, a "valid" tick, the last four digits where the site shows them),
• an error region that stayed empty,
• a button that became enabled,
• the network-level artefact of the fill itself (the tool's own record of what was typed and whether focus was placed),
• and, once submitted, a confirmation page that names the item and amount.

If none of those exist, the honest output is "filled, not verified" — not "done".

When the input never reached the page at all

Some windows are not part of the page. A JavaScript confirm, a native file chooser, an "open this in the app?" bar, a permission prompt, a passkey request — these live outside the document, so they do not appear in the text snapshot. The agent types, the page records nothing, and the transcript looks like a page that ignored a keystroke.

Repeating the key is the wrong recovery, because the key is already going somewhere: the browser's own dialog. Two practical rules:

1. When an action reports that the page did not record the input, stop and look at the pixels once. The screenshot is the only view that includes browser-level windows drawn over the page.
2. Answer the window once — accept, dismiss, or press Escape to close it and re-read the page. If the flow genuinely needs a human (a security key, a captcha), that is a handover, not a retry loop. Repeated attempts at a bot check read as a macro and get the account restricted for hours.

A related trap: a native dialog that blocks every other tool while it waits. If your harness queues actions while a confirm is open, the next three steps silently do nothing. Treat "dialog pending" as a first-class state in your loop, not as a transient hiccup.

Retry from the record, not from your log

The expensive duplicate is not caused by a flaky site. It is caused by the agent trusting its own transcript: I clicked submit, the response was lost, therefore it did not happen.

External actions persist after the response is lost. So the retry rule is:

• Read the current state before acting again. The order list, the message thread, the booking page, the balance. If the effect is there, you are done — and you may have learned that your timeout handler is lying.
• If the state is ambiguous, do not guess in the direction of action. "Cancelled" does not tell you who cancelled it or why; a status you did not set is not a failure you may overwrite.
• Never treat "already done" as an invitation to do it again with better parameters. A second submission with corrected details is a second effect, not a fix.

This is the same discipline as idempotency keys on the server side, applied at the agent layer: the key that decides whether an effect is new must be derived from the fact (action + target + the consequential inputs), not from the run that carries it. An agent that has no key at all can still approximate one by reading the target's own state first.

The money line

There is a class of action where a wrong verification is not a bug but a loss. A checkout that the user agreed to at one price returns a total with fees, service charges, or a tip added. The delta is small; the decision is not the agent's. It belongs to the person whose money it is, and they learn about it after the fact, permanently.

The rule I use: separating settled terms from new terms. If the total is what was agreed, proceed. If it differs, stop and surface the specific numbers — what was agreed, what the page wants, what the difference is, and what the alternatives are. The same applies to a deposit, a prepaid balance, a cancellation fee, and a refund that would fail silently.

An agent that verifies effects well will still be wrong about whose decision a number is. Verification and authority are separate axes; a run can be technically perfect and still cross a line.

Failure taxonomy worth encoding

Nine rows, and only one of them justifies an immediate retry.

Instrument the delta, not the intention

The log line that made this debuggable was not "clicked submit". It was:

action=submit_form target=order/9182
observed_before=status:cart  observed_after=status:confirmed ref:ORDER-9182
evidence=shot:order-confirmed.png  record=read:/account/orders#9182

Three fields matter: what you expected to change, what actually changed, and where the record can be re-read later when the screenshot is gone. Without the third field, an agent cannot answer the only question that matters after a restart: did this already happen?

Screenshots are evidence of pixels, not of state. They are still worth taking — scoped to the element when the element is the thing (a confirmation panel, a price summary), full-page when the layout is the problem — and they must be taken before navigating away, because there is no going back to that exact render. What they cannot do is replace the record.

Checklist

• Three facts per action: input reached, page changed, effect persisted. Name which one you verified.
• Read the page again before acting on it; treat every element handle as expired.
• Resolve same-name controls by the record around them, never by label alone.
• Treat a blocked click as information: dismiss the blocker, re-read, retry once.
• When the page did not record input, look at pixels once — a browser window may hold the focus.
• Verify around a masked screen without submitting it.
• Before any retry of an external action, read the target's state.
• Store an effect key derived from the fact; refuse stale writes.
• Stop at any amount the principal did not agree to, and report the numbers.
• Log the delta and a re-readable record reference, not the intention.

None of this makes a browser agent reliable. It makes the agent unable to claim an effect it did not verify, and unable to repeat one it cannot rule out — which is the only kind of reliable that survives a lost response.

If you want the concrete phone-side version of this — what the assistant does at a captcha, how a daily usage limit behaves, what a reset ticket actually restores — I keep two public notes:

• The English guide: https://pickle-invite-code-en.deed.page/ — the app's behaviour on daily limits, the 4 AM reset, and reset tickets.
• The invite link: https://pickle.com/i/RDKXK77G9ZRT

Disclosure: I use Pickle, a phone-side AI assistant app, and I am not its vendor. The two links above are mine; if a sign-up completes through the invite link, both sides receive reset tickets, and the app's own notices govern any benefit. This is not a paid placement. The verification model described here is the one I run in my own browser-operating agent, written from failures I caused myself.