# Bellman Policy Optimization

Source: [arXiv](https://arxiv.org/abs/2609.15987v1)  
Feed7 permalink: https://feed7.dev/p/2609-15987v1-1r643qk  
Published: 2026-09-14T17:59:47.000Z  
Trust: Needs Review (needs_review)

## Why Included

BPO turns policy mirror descent into a critic-free trajectory objective for terminal rewards, avoiding intermediate value estimates. It is relevant to builders training reasoning models with verifiable rewards.

## Source Summary

**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.

## Practical Implication

Teams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.

## Agent-Ready Context

**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.

Teams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.

The material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks.

## Connected Context

Feed7 judgment across 778 accumulated Signals:

BPO adds a critic-free RLVR route distinct from the supplied distillation and reward-construction methods: terminal rewards are propagated through a trajectory-level Bellman formulation rather than learned intermediate values. It may simplify reasoning-model training where critics are unstable, but the practical approximation and score-free abstract leave efficiency and comparative advantage unresolved.

- [$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation](https://feed7.dev/p/2607-28582v1-0egi1xh) — β-OPSD offers a contrasting route to stable reasoning training through tunable self-distillation without direct RL, while BPO retains RLVR but removes critic estimation.
- [RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction](https://feed7.dev/p/2608-06310v1-1e0x375) — RRC addresses how comparative judge outputs become scalar rewards; BPO addresses how terminal rewards optimize trajectories, so the mechanisms could occupy consecutive parts of one training pipeline.
- [Weak-to-Strong Generalization via Direct On-Policy Distillation](https://feed7.dev/p/2607-05394v1-0vq2jpn) — Direct-OPD turns another model’s policy change into dense student rewards, contrasting with BPO’s derivation for learning from terminal rewards without intermediate value estimates.
- [TTPO: Test-Time Policy Optimization](https://feed7.dev/p/2608-27448v1-0dysp2e) — TTPO moves policy updates to inference time using rollout agreement, whereas BPO is a training objective for terminal-reward trajectories; they represent different operational points for adapting reasoning behavior.

## Context Map

- Layer: model
- Domains: None
- Topics: reasoning

## Uncertainty

- The material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.