Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
Desktop-Delta Bench tests whether computer-use models can verify what changed after an action, exposing a reliability gap hidden by end-task and single-frame scores.
Desktop-Delta Bench contains **2,013 human-verified instances** from multi-app Linux trajectories across about 15 applications and 50 task domains. It tests temporal ordering and action inference from before-and-after observations.
Use step-level transition checks when evaluating computer-use agents, especially around stale screenshots, delayed rendering, and recovery. The best ordering exact-match rates reached only **65.1% without decoys** and **65.7% with decoys**.
Desktop-Delta Bench contains **2,013 human-verified instances** from multi-app Linux trajectories across about 15 applications and 50 task domains. It tests temporal ordering and action inference from before-and-after observations. Use step-level transition checks when evaluating computer-use agents, especially around stale screenshots, delayed rendering, and recovery. The best ordering exact-match rates reached only **65.1% without decoys** and **65.7% with decoys**. This is an offline diagnostic rather than an end-to-end task benchmark. Action recognition also varies sharply: reported click and drag scores were **0.96 versus 0.76 F1**, so aggregate GUI grounding can hide operation-specific weaknesses.
Desktop-Delta Bench narrows the evaluation gap identified by broader coding and rollout benchmarks to a specific failure surface: whether an agent understands the transition between GUI states. Its low ordering ceiling and operation-specific score spread support adding step-level temporal and action checks to reproducible trajectory evaluation, rather than treating end-to-end completion or aggregate grounding as sufficient.