{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.02817v1",
  "slug": "2609-02817v1-1ya63h2",
  "url": "https://feed7.dev/p/2609-02817v1-1ya63h2",
  "title": "Cliff: Learning Process Rewards from the First Mistake",
  "why_included": "Cliff trains on the location of a rollout’s first reasoning error, rewarding the valid prefix and penalizing the suffix. It offers finer supervision without a specialized reward model.",
  "summary": "**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.",
  "practical_implication": "For verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.",
  "agent_context": "**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.\n\nFor verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.\n\nAcross **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.02817v1",
    "published_at": "2026-09-02T17:03:42.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [],
  "topics": [
    "reasoning",
    "agent-evals"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Across **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification."
  ],
  "connected_context": {
    "meaning": "This refines process supervision from scoring whole trajectories to identifying the first invalid step, preserving credit for the valid prefix and avoiding supervision on reasoning already corrupted by an error. It complements outcome-based agent evals, but its advantage depends on a teacher reliably locating that boundary; the supplied results do not quantify this sensitivity or added cost.",
    "corpus_size": 669,
    "generated_at": "2026-09-03T10:01:29.442Z",
    "connections": [
      {
        "title": "Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.05139v1",
        "feed7_url": "https://feed7.dev/p/2608-05139v1-0ji039d",
        "reason": "Skill²-Bench identifies capability transitions as a source of long-horizon failure, while Cliff offers a complementary training signal for locating the first step where such a trajectory becomes invalid."
      },
      {
        "title": "AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.11849v1",
        "feed7_url": "https://feed7.dev/p/2607-11849v1-1sjdl4n",
        "reason": "Frontier models’ weakness at rejecting invalid proofs makes Cliff’s unreported sensitivity to teacher error consequential, because a mistaken boundary would assign token-level advantages incorrectly."
      },
      {
        "title": "RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.06310v1",
        "feed7_url": "https://feed7.dev/p/2608-06310v1-1e0x375",
        "reason": "RRC and Cliff address different reward bottlenecks: RRC preserves a judge’s relative rankings, whereas Cliff converts a detected first-error boundary into prefix-level advantages."
      },
      {
        "title": "Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.04001v1",
        "feed7_url": "https://feed7.dev/p/2608-04001v1-1gj91hk",
        "reason": "The reproducibility framework implies that Cliff’s reported gains require the full rollout, teacher, sampling, and compute protocol before they can be compared reliably with other training methods."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-02T17:03:42.000Z",
  "modified_at": "2026-09-02T17:03:42.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-02817v1-1ya63h2",
    "json": "https://feed7.dev/p/2609-02817v1-1ya63h2.json",
    "markdown": "https://feed7.dev/p/2609-02817v1-1ya63h2.md"
  }
}