# Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs

Source: [AI Engineer](https://www.youtube.com/watch?v=CTLa_p6iOiY)  
Feed7 permalink: https://feed7.dev/p/computer-use-at-the-edge-of-the-statistical-precipice-pierluca-d-oro-pro-16136hs  
Published: 2026-08-14T14:30:31.000Z  
Trust: Source Linked (source_linked)

## Why Included

Static computer-use benchmarks can reward memorized action scripts rather than adaptation. Vary task state, verify every generated case, and calculate uncertainty across both actions and environments.

## Source Summary

A replay agent stores one winning trajectory per task and blindly repeats it. On deterministic OSWorld and MobileWorld-style evaluations, a script **under 1 MB** can match or beat the frontier model that generated its traces, exposing benchmark replayability.

## Practical Implication

For agent evals, vary data, appearance, and initial state; automatically reject invalid combinations; and use privileged verifiers inside a sandbox. DGWorld applies this design across **15 apps, 387 scenarios, and 3.2 million verified configurations**.

## Agent-Ready Context

A replay agent stores one winning trajectory per task and blindly repeats it. On deterministic OSWorld and MobileWorld-style evaluations, a script **under 1 MB** can match or beat the frontier model that generated its traces, exposing benchmark replayability.

For agent evals, vary data, appearance, and initial state; automatically reject invalid combinations; and use privileged verifiers inside a sandbox. DGWorld applies this design across **15 apps, 387 scenarios, and 3.2 million verified configurations**.

Uncertainty must cover both model actions and environment variation. The talk reports that rollout-only intervals can provide roughly **17–20% coverage** where a correctly structured method approaches the intended 95%, but applying that method requires a benchmark with explicit variation structure.

## Connected Context

Feed7 judgment across 462 accumulated Signals:

This demonstrates that fixed computer-use benchmarks can measure trajectory memorization rather than general capability. It strengthens the case for varied initial states, privileged final-state verification, and uncertainty estimates that include environment variation, while limiting the proposed statistical remedy to benchmarks whose variation is explicitly structured.

- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — Supplies a concrete replay mechanism behind the broader warning that leaderboard gains can arise from contamination, weak test conditions, and shortcuts rather than useful behavior.
- [Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software](https://feed7.dev/p/rethinking-environments-for-long-horizon-work-rayan-garg-theta-software-11r7wbx) — Reinforces final-state verification but adds that difficulty and uncertainty must be measured across changing environments, not inferred from one fixed trajectory or task duration.
- [Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs](https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz) — Supports the need for repeatable yet varied simulations because agents can exploit fixed incentives and behave differently when they recognize evaluation conditions.
- [Verifiable Environments for AI in Biology — Kenny Workman, LatchBio](https://feed7.dev/p/verifiable-environments-for-ai-in-biology-kenny-workman-latchbio-1vs6y66) — Qualifies privileged deterministic verification: even strong internal graders can reject valid solution paths unless humans test the acceptable variation.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- Uncertainty must cover both model actions and environment variation. The talk reports that rollout-only intervals can provide roughly **17–20% coverage** where a correctly structured method approaches the intended 95%, but applying that method requires a benchmark with explicit variation structure.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.