# Cliff: Learning Process Rewards from the First Mistake

Source: [arXiv](https://arxiv.org/abs/2609.02817v1)  
Feed7 permalink: https://feed7.dev/p/2609-02817v1-1ya63h2  
Published: 2026-09-02T17:03:42.000Z  
Trust: Needs Review (needs_review)

## Why Included

Cliff trains on the location of a rollout’s first reasoning error, rewarding the valid prefix and penalizing the suffix. It offers finer supervision without a specialized reward model.

## Source Summary

**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.

## Practical Implication

For verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.

## Agent-Ready Context

**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.

For verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.

Across **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification.

## Connected Context

Feed7 judgment across 669 accumulated Signals:

This refines process supervision from scoring whole trajectories to identifying the first invalid step, preserving credit for the valid prefix and avoiding supervision on reasoning already corrupted by an error. It complements outcome-based agent evals, but its advantage depends on a teacher reliably locating that boundary; the supplied results do not quantify this sensitivity or added cost.

- [Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning](https://feed7.dev/p/2608-05139v1-0ji039d) — Skill²-Bench identifies capability transitions as a source of long-horizon failure, while Cliff offers a complementary training signal for locating the first step where such a trajectory becomes invalid.
- [AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification](https://feed7.dev/p/2607-11849v1-1sjdl4n) — Frontier models’ weakness at rejecting invalid proofs makes Cliff’s unreported sensitivity to teacher error consequential, because a mistaken boundary would assign token-level advantages incorrectly.
- [RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction](https://feed7.dev/p/2608-06310v1-1e0x375) — RRC and Cliff address different reward bottlenecks: RRC preserves a judge’s relative rankings, whereas Cliff converts a detected first-error boundary into prefix-level advantages.
- [Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility](https://feed7.dev/p/2608-04001v1-1gj91hk) — The reproducibility framework implies that Cliff’s reported gains require the full rollout, teacher, sampling, and compute protocol before they can be compared reliably with other training methods.

## Context Map

- Layer: model
- Domains: None
- Topics: reasoning, agent-evals

## Uncertainty

- Across **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.