Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

How to Give Your AI Coding Agent Eyes and Hands in Your Chrome Browser

Most AI coding assistants (Claude Code, Cursor, Antigravity, Codex) are trapped inside terminal sandboxes. Whenever your agent needs to check Google Search Console metrics, review a pull request on GitHub, or verify Clou

How to Give Your AI Coding Agent Eyes and Hands in Your Chrome Browser

Most AI coding assistants (Claude Code, Cursor, Antigravity, Codex) are trapped inside terminal sandboxes. Whenever your agent needs to check Google Search Console metrics, review a pull request on GitHub, or verify Cloudflare DNS settings, it hits login barriers and 2FA prompts.

Headless browsers (Puppeteer, Playwright) start with blank profiles β€” no cookies, no extensions, no developer logins. Moreover, synthetic DOM clicks (element.click()) produce event.isTrusted = false, which modern single-page applications actively ignore or block.

To solve this, we built Agent Chrome Bridge β€” an open-source bridge connecting your AI agent directly to your active desktop Chrome session via Chrome DevTools Protocol (CDP).

Architecture: Lightweight & Local-First

The system consists of three lightweight parts:

  1. Chrome Extension (Manifest V3): Attaches via chrome.debugger to dispatch engine-level CDP input events with isTrusted: true.
  2. Local Loopback Daemon (Python): Listens exclusively on 127.0.0.1:8766 (loopback only) and enforces WebSocket origin checks to prevent unauthorized access from external origins.
  3. CLI Tool (br.py): A clean CLI that agents invoke in one-line shell turns.

Agent Chrome Bridge Architecture and Visual Indexing

Visual Element Indexing: Why Numeric Tags Beat CSS Selectors

AI agents struggle with complex, obfuscated CSS selectors like button.flex.items-center.bg-blue-600.hover\:bg-blue-700.

Instead, Agent Chrome Bridge implements deterministic numeric tagging:

  1. The agent runs python br.py look.
  2. The bridge scans the visible viewport and labels interactive elements with clean numeric markers: [1], [2], [3].
  3. The agent receives a clean inventory:
   [14] button "Create repository"
   [16] input:text "Repository name"
   [18] button "Submit"
  1. The agent executes python br.py click 14 or python br.py type 16 "my-new-project".
  2. If the DOM shifts, the subsequent look recalculates indices automatically.

Real-World Example: Inspecting Google Search Console

Here is what an autonomous inspection turn looks like:

# 1. Open Google Search Console in a background tab
python br.py open https://search.google.com/search-console

# 2. Inspect visible elements
python br.py look

# 3. Click on the URL inspection search box
python br.py click 7

# 4. Type the URL and submit via trusted hardware enter
python br.py type 7 "https://example.com/blog/my-post/" --enter

# 5. Capture an annotated screenshot to verify results
python br.py shot

The entire flow takes seconds and reuses your existing Google login session without prompting for 2FA or exposing session tokens to cloud proxies.

Security Boundaries: Strict Rules for AI Agents

Granting an AI agent browser capabilities requires disciplined guardrails:

  • Zero Credentials in LLM Prompts: The agent must never be asked to enter master passwords, API keys, or credit card details. The human logs in; the agent acts within the authenticated tab.
  • Human-in-the-Loop for Payments & CAPTCHAs: The bridge does not solve CAPTCHAs or complete payment forms. When encountering a gate, it pauses and notifies the user.
  • Local Isolation: The daemon runs on loopback 127.0.0.1 only, and the extension validates message origin.

Getting Started

The project is completely free and open source:

Try it with your AI coding agent and let us know what you think in the comments!

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.