Computer Use

Cursor cloud agents can now execute tools on machines and elastic pools inside your network while Cursor retains planning and inference. This unlocks internal access and custom hardware, not full self-hosting.

SAGE uses an expensive VLM only when a learner is uncertain, tests its advice against environment rewards, and distills useful guidance into a policy that needs no VLM at runtime.

Web agents should validate fetched content before spending model tokens on it and reserve browser automation for interactive steps. Explicit failures beat silently passing CAPTCHAs or blocked HTML downstream.

Computer-use agents fail after ordinary UI surprises: expired sessions, deceptive buttons and persistent side effects. Safer deployments need realistic training, risk-aware harnesses and explicit user handoff.

Reliable browser agents pair a programmable CLI with Chrome DevTools Protocol input, then sense, act, and verify one move at a time. Save solved paths as deterministic code instead of re-reasoning.

Production browser agents need more than a capable model: a domain harness, compact site knowledge, consistent rendering, secure authentication, and human approval for sensitive actions.

The long-tail web is unlikely to expose clean agent APIs. Browser agents need pixels as ground truth, with code and network access used opportunistically for speed rather than as universal substitutes.

Kenton Varda argues that personal AI-generated apps need per-user code and strong isolation, not one server-owned version. The demo shows agents modifying app code inside a constrained local runtime.

MiniMax and Together describe why agent workloads change inference priorities: large codebase context, repeated tool turns, multimodality, and long-horizon RL all demand purpose-built environments and kernels.

OSReward finds that VLM judges often approve failed computer-use runs. Its benchmark and open reward models offer a more grounded way to evaluate trajectories without paying frontier-model costs.

Desktop-Delta Bench tests whether computer-use models can verify what changed after an action, exposing a reliability gap hidden by end-task and single-frame scores.

Persistent Codex workflows become more useful with reusable skills, memory, app-aware context, and scheduled thread check-ins—but computer use needs explicit boundaries and stopping rules.

OpenForgeRL trains agents inside their deployment harnesses by proxying model calls into standard RL stacks and isolating rollouts in Kubernetes containers; error recovery remains weak.

Perception agents share the rendered interface with users, turning visual annotations and design rules into agent input and checks. The open-source harness is an early attempt to close computer-use feedback loops.

Experimental on-device agents can play games and adapt interfaces without cloud calls, but real-time use must fit memory, frame-time, and battery budgets. Accessibility is promising, not production-ready.

PalmClaw runs the agent loop, memory, skills, and tools directly on a phone, exposing device capabilities as structured calls instead of GUI gestures. The paper reports faster, more reliable task execution.

A free, code-oriented Chinese curriculum spans model tuning, deployment, agents, alignment, security, and multimodal systems. It is useful as a broad learning map, but remains a work in progress.

Sonnet 5 lands June 30 at $2/$10 per Mtok intro pricing (through Aug 31), nearing Opus 4.8 on agentic coding and computer use. Note the new tokenizer: inputs map to 1.0–1.35x more tokens.

Gemini Spark lands on macOS (US, AI Ultra beta) and gains custom MCP support plus connectors for Tasks, Keep, Canva and Dropbox — Google's assistant now speaks the protocol your agent tooling already uses.

Alibaba's page-agent is an MIT-licensed JS library that embeds a natural-language GUI agent in any webpage via one script tag. It drives the DOM as text — no extension, headless browser, or multimodal model needed.

Chrome DevTools MCP gives coding agents direct access to browser automation, network data, console errors, screenshots, and performance traces, with privacy controls worth setting explicitly.

Google's June roundup: Gemma 4 12B runs locally in 16GB of memory, Gemini 3.5 Flash adds computer use for desktop, mobile, and browser agents, and Nano Banana 2 Lite ships as a cheaper image model.

ego lite lets Codex, Claude Code, and other agents automate logged-in web sessions in isolated browser spaces without taking over your active tabs. It is macOS-only today.