{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.15987v1",
  "slug": "2609-15987v1-1r643qk",
  "url": "https://feed7.dev/p/2609-15987v1-1r643qk",
  "title": "Bellman Policy Optimization",
  "why_included": "BPO turns policy mirror descent into a critic-free trajectory objective for terminal rewards, avoiding intermediate value estimates. It is relevant to builders training reasoning models with verifiable rewards.",
  "summary": "**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.",
  "practical_implication": "Teams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.",
  "agent_context": "**Bellman Policy Optimization** derives a critic-free RLVR objective from Policy Mirror Descent. For autoregressive generation with terminal rewards, Bellman equations move optimization to the trajectory level and remove intermediate state-value estimation.\n\nTeams training reasoning models can evaluate BPO when critic estimation adds complexity or instability. Its practical loss uses a smoothed ratio of complementary token probabilities to correct mismatch.\n\nThe material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.15987v1",
    "published_at": "2026-09-14T17:59:47.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [],
  "topics": [
    "reasoning"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The material reports effectiveness on **mathematical reasoning benchmarks** but provides no scores or comparisons. The practical loss approximates the theoretical objective, so the abstract alone does not establish its cost or behavior outside terminal-reward tasks."
  ],
  "connected_context": {
    "meaning": "BPO adds a critic-free RLVR route distinct from the supplied distillation and reward-construction methods: terminal rewards are propagated through a trajectory-level Bellman formulation rather than learned intermediate values. It may simplify reasoning-model training where critics are unstable, but the practical approximation and score-free abstract leave efficiency and comparative advantage unresolved.",
    "corpus_size": 778,
    "generated_at": "2026-09-15T10:07:01.848Z",
    "connections": [
      {
        "title": "$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.28582v1",
        "feed7_url": "https://feed7.dev/p/2607-28582v1-0egi1xh",
        "reason": "β-OPSD offers a contrasting route to stable reasoning training through tunable self-distillation without direct RL, while BPO retains RLVR but removes critic estimation."
      },
      {
        "title": "RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.06310v1",
        "feed7_url": "https://feed7.dev/p/2608-06310v1-1e0x375",
        "reason": "RRC addresses how comparative judge outputs become scalar rewards; BPO addresses how terminal rewards optimize trajectories, so the mechanisms could occupy consecutive parts of one training pipeline."
      },
      {
        "title": "Weak-to-Strong Generalization via Direct On-Policy Distillation",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.05394v1",
        "feed7_url": "https://feed7.dev/p/2607-05394v1-0vq2jpn",
        "reason": "Direct-OPD turns another model’s policy change into dense student rewards, contrasting with BPO’s derivation for learning from terminal rewards without intermediate value estimates."
      },
      {
        "title": "TTPO: Test-Time Policy Optimization",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.27448v1",
        "feed7_url": "https://feed7.dev/p/2608-27448v1-0dysp2e",
        "reason": "TTPO moves policy updates to inference time using rollout agreement, whereas BPO is a training objective for terminal-reward trajectories; they represent different operational points for adapting reasoning behavior."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-14T17:59:47.000Z",
  "modified_at": "2026-09-14T17:59:47.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-15987v1-1r643qk",
    "json": "https://feed7.dev/p/2609-15987v1-1r643qk.json",
    "markdown": "https://feed7.dev/p/2609-15987v1-1r643qk.md"
  }
}