arXivPaperNeeds Review

Bellman Policy Optimization

BPO turns policy mirror descent into a critic-free trajectory objective for terminal rewards, avoiding intermediate value estimates. It is relevant to builders training reasoning models with verifiable rewards.

arXiv · Sep 14, 2026

Open Source Open Markdown Open JSON

Source Summary

**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.

Practical Implication

Teams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.

Agent-Ready Context

**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.

Teams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.

The material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks.

Connected Context · Feed7 Judgment

BPO adds a critic-free RLVR route distinct from the supplied distillation and reward-construction methods: terminal rewards are propagated through a trajectory-level Bellman formulation rather than learned intermediate values. It may simplify reasoning-model training where critics are unstable, but the practical approximation and score-free abstract leave efficiency and comparative advantage unresolved.

$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillationβ-OPSD offers a contrasting route to stable reasoning training through tunable self-distillation without direct RL, while BPO retains RLVR but removes critic estimation.RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward ConstructionRRC addresses how comparative judge outputs become scalar rewards; BPO addresses how terminal rewards optimize trajectories, so the mechanisms could occupy consecutive parts of one training pipeline.Weak-to-Strong Generalization via Direct On-Policy DistillationDirect-OPD turns another model’s policy change into dense student rewards, contrasting with BPO’s derivation for learning from terminal rewards without intermediate value estimates.TTPO: Test-Time Policy OptimizationTTPO moves policy updates to inference time using rollout agreement, whereas BPO is a training objective for terminal-reward trajectories; they represent different operational points for adapting reasoning behavior.

Context Map

model#reasoning

Uncertainty

The material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks.