# When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning

Source: [arXiv](https://arxiv.org/abs/2607.29617v1)  
Feed7 permalink: https://feed7.dev/p/2607-29617v1-0zokmcv  
Published: 2026-07-31T16:52:47.000Z  
Trust: Needs Review (needs_review)

## Why Included

OVI suggests interactive expert feedback can let a smaller learner imitate values without representing the expert's full policy. The result matters to agent distillation, but relies on specific assumptions.

## Source Summary

The paper introduces **OVI**, an interactive on-policy imitation-learning method. Its central result is that expert interaction can reduce the learner's representational burden from matching the expert policy to realizing the expert's value function.

## Practical Implication

For distilling agent behavior into a less expressive model, reconsider a demonstration-only pipeline: querying the expert along learner-generated trajectories may expose corrective evidence where errors actually accumulate. OVI reportedly beats **BC**, **DAgger**, and offline value-based methods, with its largest gains for smaller learners.

## Agent-Ready Context

The paper introduces **OVI**, an interactive on-policy imitation-learning method. Its central result is that expert interaction can reduce the learner's representational burden from matching the expert policy to realizing the expert's value function.

For distilling agent behavior into a less expressive model, reconsider a demonstration-only pipeline: querying the expert along learner-generated trajectories may expose corrective evidence where errors actually accumulate. OVI reportedly beats **BC**, **DAgger**, and offline value-based methods, with its largest gains for smaller learners.

The guarantees require expert-value realizability and, for computational efficiency, access to a linear maximization oracle. The negative result says offline learning must otherwise scale with expert-policy complexity, but the supplied abstract gives no effect sizes or deployment evidence for coding agents.

## Connected Context

Feed7 judgment across 330 accumulated Signals:

This sharpens the case for on-policy agent distillation: corrective expert queries on learner-generated trajectories can reduce what a smaller learner must represent, rather than merely adding more demonstrations. It reinforces trajectory compatibility as important for long-horizon learning and gives a theoretical reason smaller learners may benefit most, but its guarantees and oracle requirement limit direct conclusions for deployed coding agents.

- [The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation](https://feed7.dev/p/2607-24720v1-0gihy13) — Both support training on learner-relevant trajectories rather than isolated demonstrations; OVI adds a representational account of why on-policy expert interaction can help less expressive learners.
- [Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect](https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9) — Production-trace mining identifies where deployed agents fail, while OVI provides a complementary method for querying an expert along those learner-generated failure trajectories; the former still requires proxy-reward safeguards.

## Context Map

- Layer: agent
- Domains: research
- Topics: agent-reliability

## Uncertainty

- The guarantees require expert-value realizability and, for computational efficiency, access to a linear maximization oracle. The negative result says offline learning must otherwise scale with expert-policy complexity, but the supplied abstract gives no effect sizes or deployment evidence for coding agents.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.