From RL to IRL — Gaurav Mishra, Amazon AGI Lab
Computer-use agents fail after ordinary UI surprises: expired sessions, deceptive buttons and persistent side effects. Safer deployments need realistic training, risk-aware harnesses and explicit user handoff.
In one expense task, an agent guessed an expired password twice and locked the account; in another, it clicked a sponsored submit button and began entering personal data elsewhere. The failures expose **partial observability**, **ambiguous completion** and adversarial interface content.
Train and evaluate recovery as a native behavior: wait, compare, backtrack, abandon or escalate. At runtime, add **checkpointing and rollback**, credential guards, action-risk classification, execution monitoring and audit logs rather than relying only on an end-state verifier.
In one expense task, an agent guessed an expired password twice and locked the account; in another, it clicked a sponsored submit button and began entering personal data elsewhere. The failures expose **partial observability**, **ambiguous completion** and adversarial interface content. Train and evaluate recovery as a native behavior: wait, compare, backtrack, abandon or escalate. At runtime, add **checkpointing and rollback**, credential guards, action-risk classification, execution monitoring and audit logs rather than relying only on an end-state verifier. The improved trajectory recognized both the sponsored control and expired session, handed authentication to the user, then resumed with preserved state. It is still a training example, not evidence that these safeguards cover arbitrary websites or irreversible actions.
This makes recovery and safe interruption first-class computer-use behaviors rather than exceptions handled by an end-state verifier. Against prior harness proposals, it supplies concrete failure modes—credential lockout, deceptive controls, lost sessions—and a runtime stack of checkpoints, rollback, risk classification, monitoring, and user handoff. It confirms the weakness of success-only evaluation while leaving generalization to arbitrary sites and irreversible actions unresolved.