When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
OVI suggests interactive expert feedback can let a smaller learner imitate values without representing the expert's full policy. The result matters to agent distillation, but relies on specific assumptions.
The paper introduces **OVI**, an interactive on-policy imitation-learning method. Its central result is that expert interaction can reduce the learner's representational burden from matching the expert policy to realizing the expert's value function.
For distilling agent behavior into a less expressive model, reconsider a demonstration-only pipeline: querying the expert along learner-generated trajectories may expose corrective evidence where errors actually accumulate. OVI reportedly beats **BC**, **DAgger**, and offline value-based methods, with its largest gains for smaller learners.
The paper introduces **OVI**, an interactive on-policy imitation-learning method. Its central result is that expert interaction can reduce the learner's representational burden from matching the expert policy to realizing the expert's value function. For distilling agent behavior into a less expressive model, reconsider a demonstration-only pipeline: querying the expert along learner-generated trajectories may expose corrective evidence where errors actually accumulate. OVI reportedly beats **BC**, **DAgger**, and offline value-based methods, with its largest gains for smaller learners. The guarantees require expert-value realizability and, for computational efficiency, access to a linear maximization oracle. The negative result says offline learning must otherwise scale with expert-policy complexity, but the supplied abstract gives no effect sizes or deployment evidence for coding agents.
This sharpens the case for on-policy agent distillation: corrective expert queries on learner-generated trajectories can reduce what a smaller learner must represent, rather than merely adding more demonstrations. It reinforces trajectory compatibility as important for long-horizon learning and gives a theoretical reason smaller learners may benefit most, but its guarantees and oracle requirement limit direct conclusions for deployed coding agents.