Browser agent benchmark, round 2: BrowserOS, Claude in Chrome, 5dive, OpenClaw, Hermes and Grok Bot on 100 live tasks
the top browser agents still finish about the same share of live tasks. this round, what set them apart was how they recover when a normal click fails. round 1 ran five setups. round 2 runs the same 100 Online-Mind2Web
the top browser agents still finish about the same share of live tasks. this round, what set them apart was how they recover when a normal click fails.
round 1 ran five setups. round 2 runs the same 100 Online-Mind2Web tasks on live websites and adds BrowserOS and Grok Bot. the six local setups ran on the same datacenter server with Claude Haiku 5.5, headed browsers, a 10-minute cap and one attempt, and the benchmark's own judge scored them.
100 live websites
- BrowserOS (Claude Code): 40 of 100
- Claude in Chrome: 37
- 5dive browser (Claude Code): 34
- OpenClaw browser: 29
- 5dive browser (pi): 28
- Hermes browser: 27
- Grok Bot: 18, on its own cloud and model, with a different judge
that judge, WebJudge on GPT-6 Luna, sees each run's actions and screenshots, never the final answer. the gaps between the top three aren't statistically significant (BrowserOS vs 5dive browser, p = 0.29).
why BrowserOS came first
it had more ways to recover when a normal click failed: 94 clicks by screen position and 216 JavaScript calls on the page. 6% of its tool calls failed, against 19% for ours. our browser can't do those fallbacks yet, and we're adding them next.
the judge matters
our earlier judge read final answers and scored BrowserOS 50, 5dive browser 39. the official judge gave them 40 and 34. on the 27 hardest tasks, every agent finished just 2 to 4.
why pi cost less than Claude Code
pi made 20% fewer tool calls and started each call with far less context: about 5.6K tokens against 17.7K. cost per finished task: 2.8Β’ against 6.1Β’.
pi also scored lower. it often pulls page text with grep and python, which leaves fewer screenshots for the judge, so some answers that look right weren't credited.
what didn't help
- our browser fixes since round 1 made it faster, and success barely moved. with the same judge it went from 32 to 33 passes, and from 9.1 to 7.9 seconds per browser step. the target is under 5 seconds.
- about 20% of runs hit a bot wall, and seven tasks blocked every agent. a residential IP barely helped in an earlier test.
- Sonnet didn't beat Haiku in an earlier test. on 62 tasks Sonnet 5.5 passed 10 and Haiku 15, at 10 times the cost per finished task (p = 0.30).
4 everyday tasks
three form edits on a test app and one Wikipedia lookup, 3 runs each. all six local setups passed 12 of 12, and Grok Bot passed 11.
BrowserOS was fastest at 19.1 seconds. 5dive browser (pi) was 0.7 seconds behind, at about an eighth of the cost ($0.82 per 1,000 tasks) and with fewer steps (4 against 9.5).
limits
- one run per task, on live sites, scored by automated judges, on a rig we built.
- Grok Bot used its own model and a different judge.
- these aren't official leaderboard results.
the full benchmark has all 700 runs and the judge's notes. we'll run round 3 in two weeks, with BrowserOS-style fallbacks in the 5dive browser.
run it yourself
the 5dive browser is free and Apache 2.0 licensed.
curl -fsSL https://install.5dive.ai | sudo bash
sudo 5dive plugin add 5dive-ai/5dive-browser
sudo 5dive browser setup
sudo 5dive init # your first agent, after setup so it gets the browser
use github.com/5dive-ai/5dive to self-host.
use 5dive.ai for a managed box.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.