AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control
AgentRoboCritic grades robot actions against each agent’s stated intent, exposing failures that episode-level success rates miss. Its released critic improves action failure-mode accuracy on a human-annotated set.
The paper analyzes direct robot control across **12 manipulation benchmarks** with a critic that compares each action’s outcome to the policy agent’s stated local intent. It releases **12,844 annotated actions**; a fine-tuned open-source critic reaches **74.24%** failure-mode accuracy, up from 32.58%.
For builders evaluating agents that act through tools or devices, action-level critiques may help locate where execution breaks down instead of relying only on task completion. The paper also reports recovery from some local failures and zero-shot manipulation without task-specific adaptation.
The paper analyzes direct robot control across **12 manipulation benchmarks** with a critic that compares each action’s outcome to the policy agent’s stated local intent. It releases **12,844 annotated actions**; a fine-tuned open-source critic reaches **74.24%** failure-mode accuracy, up from 32.58%. For builders evaluating agents that act through tools or devices, action-level critiques may help locate where execution breaks down instead of relying only on task completion. The paper also reports recovery from some local failures and zero-shot manipulation without task-specific adaptation. The evaluation is about robot manipulation, with physical limits around grasping and alignment and uneven recovery across failure modes. The critic’s accuracy was validated on a **400-action human-annotated benchmark**; broader applicability is not established here.
AgentRoboCritic evaluates robot-control actions against the policy’s stated local intent, making action-level failure analysis a complement to task-level outcomes. Its annotated manipulation data and measured critic accuracy support that method in the tested robotics setting, but do not establish transfer to other tool-using agents.