arXivPaperNeeds Review

TTPO: Test-Time Policy Optimization

TTPO adapts a model at test time without labels by treating agreement and disagreement differently, reducing the damage from wrong majority-vote pseudo-labels.

arXiv

Open Source Open Markdown Open JSON

Source Summary

TTPO uses majority votes as pseudo-labels but avoids trusting them uniformly. It distills agreeing rollouts with on-policy self-distillation and penalizes disagreeing rollouts with grouped reinforcement learning, while token selection down-weights converged positions and targets confident errors.

Practical Implication

Without labels, **TTPO matched label-supervised OPSD on five benchmarks**. In test-time training it raised **Qwen3-1.7B from 38.0% to 45.2%**, suggesting adaptive inference can improve a small model without an external answer key.

Agent-Ready Context

TTPO uses majority votes as pseudo-labels but avoids trusting them uniformly. It distills agreeing rollouts with on-policy self-distillation and penalizes disagreeing rollouts with grouped reinforcement learning, while token selection down-weights converged positions and targets confident errors.

Without labels, **TTPO matched label-supervised OPSD on five benchmarks**. In test-time training it raised **Qwen3-1.7B from 38.0% to 45.2%**, suggesting adaptive inference can improve a small model without an external answer key.

The evidence is limited to competition-level mathematical reasoning in the supplied abstract. Test-time parameter updates also differ from ordinary agent retries or critique loops, so applicability to coding agents and the operational cost of per-task adaptation remain open questions.

Context Map

modelresearch#reasoning

Uncertainty

The evidence is limited to competition-level mathematical reasoning in the supplied abstract. Test-time parameter updates also differ from ordinary agent retries or critique loops, so applicability to coding agents and the operational cost of per-task adaptation remain open questions.