The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore
Reliable browser agents pair a programmable CLI with Chrome DevTools Protocol input, then sense, act, and verify one move at a time. Save solved paths as deterministic code instead of re-reasoning.
The approach gives agents a CLI over Chrome DevTools Protocol, exposing the DOM, accessibility tree, screenshots, network traffic, logs, clicks, and keystrokes. Its core control loop is **sense, act, verify** across independent channels.
Start with cheap programmatic actions, escalate to trusted browser input only when required, and use vision or human-like motion as the last rung. Once a path works, encode it so deterministic code handles repeated interaction while the model handles perception.
The approach gives agents a CLI over Chrome DevTools Protocol, exposing the DOM, accessibility tree, screenshots, network traffic, logs, clicks, and keystrokes. Its core control loop is **sense, act, verify** across independent channels. Start with cheap programmatic actions, escalate to trusted browser input only when required, and use vision or human-like motion as the last rung. Once a path works, encode it so deterministic code handles repeated interaction while the model handles perception. The demonstrations include bypassing **Turnstile, MTCaptcha, Lemon, and reCAPTCHA v2**, behavior that can trigger abuse controls and violate site policies. The method therefore needs explicit authorization and tight operational boundaries.
This makes browser automation a layered harness problem: combine independent observations, verify each action, and convert successful paths into deterministic routines. It complements visual-only feedback with DOM, accessibility, network, and log evidence, while its CAPTCHA demonstrations sharply increase the need for authorization, action boundaries, and approval controls.