How to compare AI assistants that actually click from chatbots that only talk

Ask ten people to compare AI assistants and you get ten feature lists: model names, context windows, plugin counts, voices. None of those predict whether the thing will book the appointment, fill the form, or cancel the subscription you keep forgetting. This post is a comparison method for the difference that matters — whether the assistant ends with a changed state in the world, and whether you can verify it.

I write this as a user of one of these tools, not as a vendor. The framework below is what I use to decide where an assistant goes in my own work, and it applies whether you are evaluating a chat product, a browser-agent startup, or a phone-side app that operates your own device.

First split: four kinds of "AI assistant"

Most bad comparisons come from putting four different products in one basket. They fail differently, cost differently, and are trustworthy for different things.

The order matters. Reach that is supported by an API is repeatable; reach that depends on reading a screen is probabilistic. A browser agent that clicks through a web console is not doing the same job as a plugin calling the same service — different error rates, different evidence, different blast radius when it is wrong.

If your candidate assistant is chat-only, "does it do tasks" is not a question you can ask; it cannot. That is not a defect, it is a category. The defect is when the product's marketing blurs the category, and you only find out at the confirmation screen.

Second split: what happens at a wall

The moment that separates a demo from a product is not the happy path. It is the first screen the assistant is not allowed to pass: a one-time code, a passkey prompt, a CAPTCHA, a 3-D Secure challenge, an identity verification, a "confirm your card" page.

The wall exists because a person is supposed to be present. Two things to check when comparing tools:

Does it stop, or does it claim victory? A tool that says "done" when the OTP screen is on screen is worse than one that says "stopped at verification, here is the code field, I have 40 seconds left". You want the stop to be a named state, not a generic error.

Does it hand back a resumable task? The useful handover tells you where it stopped, what is already committed (was the booking held? was the form submitted?), what is still open, and what it needs from you. A tool that hands you a screenshot and no state has offloaded the diagnosis to you, which is most of the work.

Passing those walls silently is not a feature to reward. An assistant that solves CAPTCHAs, reads your SMS for codes, or drives the passkey prompt is either breaking the service's terms or making it impossible to tell what actually happened. On a comparison sheet, treat "bypasses verification" as a red flag, not a headline.

Third split: how it proves it did the thing

Ask any assistant for the same information after a task: what changed, where can I see it, and what did you skip. The answer quality is the cheapest test you can run, and it separates products more sharply than any benchmark.

Good — "Booked table for 2 at 19:00 Thursday, confirmation A17-4432, I skipped adding a note field because it required a login. Nothing else was touched."

Bad — "I've successfully booked your table! Let me know if you need anything else."

The second answer is not a lie you can catch later. Both are generated text; only one is checkable. When you are comparing tools, run one small task with a deliberately ambiguous instruction — "sort out my dentist thing" — and see whether the tool asks one short clarifying question, guesses and commits, or guesses and reports success. Guessing and committing is the failure mode that costs money. Guessing and saying so is acceptable. Asking is best.

Two implementation details worth checking in the product's own documentation, because they predict real-world behaviour:

Does it verify state after an action? Clicking is an intention. Whether the click did anything is a second observation. A tool without that second step will eventually submit a form twice, and duplicate bookings and duplicate invoices are expensive to unwind.

Is the target identified by meaning or by coordinates? Anything that stores pixel positions or DOM paths breaks on a layout change or a promotion banner. Anything that re-reads the page and finds the element again survives it. You cannot see this from a demo. You can see it from a changelog and from whether the tool's failure messages mention elements rather than coordinates.

The comparison table I actually use

Fill this in for each candidate, with one real task of your own — not the vendor's demo.

Cost deserves a note, because the categories price differently and the difference is not only money. Token-metered tools make you think about length. Subscription tools make you use them enough to justify the fee. A daily quota with a reset makes you ration the same way you ration any scarce resource — and, importantly, it caps your worst day instead of surprising you at the end of the month. If a tool has a usage limit, find out exactly when it resets, what happens to unused capacity, and whether anything can restore it (an invite, a purchase, waiting). Those three facts change your workflow more than the model version does.

A five-task protocol you can run in an evening

Pick tasks you can verify in under a minute each, and one that fails on purpose.

1. A read that needs a login. Ask it to tell you the balance or the next appointment from an account that requires signing in. Watch whether it reports what it read or what it expected to read.
2. A write with a reversible outcome. Create a calendar event in a free slot. Then check the calendar yourself. This tests state verification cheaply.
3. A write with a wall. Book something that requires a verification step, and stop at the wall. Score the handover: did it name the wall, say what was committed, and give you a resumable next action?
4. A task with an ambiguous instruction. See whether it asks, or commits and reports success.
5. A task you cancel halfway. Interrupt it mid-flow and see if it stops cleanly or leaves a half-submitted form.

Score each of the five zero, one, or two — zero for claiming success it cannot show, one for a clean stop with state, two for a verified result. Anything that scores zero twice on task 3 or 4 is not ready to touch accounts or money, whatever its benchmark numbers say.

Add one more check if the tool runs on your phone rather than a server: what can it see? A phone-side agent can reach things no browser session can, which is exactly why its read scope is the biggest privacy decision in the purchase. Look for a stated scope (which apps, which data), a way to narrow it, and whether it can be told to stop before an action instead of after.

Where this framework points

If your work is words, a chat product is the correct purchase and the comparison is about output quality. If your work is inside services with APIs, a tool-calling assistant wins on reliability. If your work is a long tail of websites, a browser agent is the only category that reaches it, and you should budget for its failure rate in your routine. And if your admin exists mainly as apps on your phone — the appointments, the deliveries, the forms that only work in an app — a phone-side agent is the only one that touches it at all, with the corresponding privacy trade-off.

The question worth asking is not "how smart is it". It is: when it stops, does it leave me something I can finish in ten seconds, and can I see what it changed? Everything else in a feature list is downstream of that.

Disclosure: I'm a user of Pickle, a Korean phone-side AI agent app. The links below are mine: an invite link that gives both sides reset tickets for the app's daily usage limit if a sign-up completes through it, and a comparison page I wrote on where conversational, browser and phone-side assistants stop. Nothing in this post depends on them — read it, skip the links, or take only the comparison method.

• Where each kind of AI assistant stops — conversational, browser and phone-side, compared
• English guide to the app, its daily reset and its limits
• Invite link: https://pickle.com/i/RDKXK77G9ZRT — 30 slots per link, first come; the app's own notices govern the benefits.