# The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

Source: [AI Engineer](https://www.youtube.com/watch?v=26RtyAm9y_Q)  
Feed7 permalink: https://feed7.dev/p/the-dark-arts-of-web-automation-teaching-agents-to-use-websites-like-hum-12nj0e0  
Published: 2026-08-14T15:30:00.000Z  
Trust: Source Linked (source_linked)

## Why Included

Reliable browser agents pair a programmable CLI with Chrome DevTools Protocol input, then sense, act, and verify one move at a time. Save solved paths as deterministic code instead of re-reasoning.

## Source Summary

The approach gives agents a CLI over Chrome DevTools Protocol, exposing the DOM, accessibility tree, screenshots, network traffic, logs, clicks, and keystrokes. Its core control loop is **sense, act, verify** across independent channels.

## Practical Implication

Start with cheap programmatic actions, escalate to trusted browser input only when required, and use vision or human-like motion as the last rung. Once a path works, encode it so deterministic code handles repeated interaction while the model handles perception.

## Agent-Ready Context

The approach gives agents a CLI over Chrome DevTools Protocol, exposing the DOM, accessibility tree, screenshots, network traffic, logs, clicks, and keystrokes. Its core control loop is **sense, act, verify** across independent channels.

Start with cheap programmatic actions, escalate to trusted browser input only when required, and use vision or human-like motion as the last rung. Once a path works, encode it so deterministic code handles repeated interaction while the model handles perception.

The demonstrations include bypassing **Turnstile, MTCaptcha, Lemon, and reCAPTCHA v2**, behavior that can trigger abuse controls and violate site policies. The method therefore needs explicit authorization and tight operational boundaries.

## Connected Context

Feed7 judgment across 479 accumulated Signals:

This makes browser automation a layered harness problem: combine independent observations, verify each action, and convert successful paths into deterministic routines. It complements visual-only feedback with DOM, accessibility, network, and log evidence, while its CAPTCHA demonstrations sharply increase the need for authorization, action boundaries, and approval controls.

- [Perception Agents — Antje Barth, Amazon AGI Lab](https://feed7.dev/p/perception-agents-antje-barth-amazon-agi-lab-1gq6jg4) — The perception harness supplies one feedback channel; this Signal broadens verification across visual, DOM, accessibility, network, and log evidence.
- [From RL to IRL — Gaurav Mishra, Amazon AGI Lab](https://feed7.dev/p/from-rl-to-irl-gaurav-mishra-amazon-agi-lab-0ex196x) — Its sense-act-verify loop addresses ordinary interaction failures, while the candidate adds checkpoints, rollback, interruption, and handoff for persistent or risky consequences.
- [Bringing agents onto the world wide web — Paul Klein IV, Browserbase](https://feed7.dev/p/bringing-agents-onto-the-world-wide-web-paul-klein-iv-browserbase-1vea3od) — Both place browser reliability in the runtime harness; the candidate adds stable environments, authentication, site knowledge, and approvals needed around this interaction ladder.
- [In Code They Act, In Proof We Trust — Erik Meijer, Leibniz Labs](https://feed7.dev/p/in-code-they-act-in-proof-we-trust-erik-meijer-leibniz-labs-1lyno2y) — Encoding proven paths as deterministic code makes actions more inspectable, but the candidate imposes the stronger prerequisite of machine-checkable safety before side effects execute.

## Context Map

- Layer: agent
- Domains: coding
- Topics: computer-use, harness-engineering, tool-use

## Uncertainty

- The demonstrations include bypassing **Turnstile, MTCaptcha, Lemon, and reCAPTCHA v2**, behavior that can trigger abuse controls and violate site policies. The method therefore needs explicit authorization and tight operational boundaries.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.