# Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs

Source: [AI Engineer](https://www.youtube.com/watch?v=CTLa_p6iOiY)  
Feed7 permalink: https://feed7.dev/p/computer-use-at-the-edge-of-the-statistical-precipice-pi-a24f8f74a4  
Published: 2026-08-14T00:00:00.000Z  
Trust: Source Linked (source_linked)

## Why Included

Vary task data, appearance, and initial state so agent evals measure adaptation instead of rewarding replayed action scripts.

## Source Summary

Static computer-use benchmarks can reward memorized action scripts rather than adaptation. Vary task state, verify every generated case, and calculate uncertainty across both actions and environments.

## Practical Implication

For agent evals, vary data, appearance, and initial state; automatically reject invalid combinations; and use privileged verifiers inside a sandbox. DGWorld applies this design across 15 apps, 387 scenarios, and 3.2 million verified configurations.

## Agent-Ready Context

A replay agent stores one winning trajectory per task and blindly repeats it. On deterministic OSWorld and MobileWorld-style evaluations, a script **under 1 MB** can match or beat the frontier model that generated its traces, exposing benchmark replayability.

For agent evals, vary data, appearance, and initial state; automatically reject invalid combinations; and use privileged verifiers inside a sandbox. DGWorld applies this design across **15 apps, 387 scenarios, and 3.2 million verified configurations**.

Uncertainty must cover both model actions and environment variation. The talk reports that rollout-only intervals can provide roughly **17–20% coverage** where a correctly structured method approaches the intended 95%, but applying that method requires a benchmark with explicit variation structure.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- None recorded.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.