Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy
An EBench study finds that a robot-control agent can complement post-trained action policies: a harness that invokes them matched the strongest compared policies while using fewer tokens. Invocation and handoff still limit performance.
On EBench, GPT-6-Astra was evaluated with direct waypoint control and with a harness that could invoke a post-trained VLA or WAM policy. Direct control led on short-horizon tasks but lagged on long-horizon, tabletop, and high-precision contact tasks. The harness configuration reached the strongest compared policies’ overall task-success level while reducing token cost.
For agent builders, this supports designing a harness that delegates execution to a specialized policy when needed. The paper identifies policy selection, missing subgoals, and disrupted grasps after control returns to the agent as current failure points.
On EBench, GPT-6-Astra was evaluated with direct waypoint control and with a harness that could invoke a post-trained VLA or WAM policy. Direct control led on short-horizon tasks but lagged on long-horizon, tabletop, and high-precision contact tasks. The harness configuration reached the strongest compared policies’ overall task-success level while reducing token cost. For agent builders, this supports designing a harness that delegates execution to a specialized policy when needed. The paper identifies policy selection, missing subgoals, and disrupted grasps after control returns to the agent as current failure points. The reported setup is robot manipulation under a shared observation protocol. Performance still fell short of the per-task upper bound of the standalone components, and these findings may not transfer to other agent tasks.
This shows that delegating execution from a general agent to a specialized manipulation policy can improve long-horizon and precise-contact performance while lowering token cost, though handoffs and policy choice remain failure points. It makes harness coordination itself part of task success and narrows the evidence to robot manipulation under the reported protocol.