arXivPaperNeeds Review

Cliff: Learning Process Rewards from the First Mistake

Cliff trains on the location of a rollout’s first reasoning error, rewarding the valid prefix and penalizing the suffix. It offers finer supervision without a specialized reward model.

arXiv · Sep 2, 2026

Open Source Open Markdown Open JSON

Source Summary

**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.

Practical Implication

For verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.

Agent-Ready Context

**Cliff** asks an off-the-shelf teacher model to find the first mistake in each rollout. It converts the split into token-level advantages: positive for the correct prefix and negative after the error.

For verifiable agent tasks, capture where a trajectory first becomes invalid instead of assigning only a final pass/fail reward. This avoids spending supervision on reasoning already conditioned on a broken prefix.

Across **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification.

Connected Context · Feed7 Judgment

This refines process supervision from scoring whole trajectories to identifying the first invalid step, preserving credit for the valid prefix and avoiding supervision on reasoning already corrupted by an error. It complements outcome-based agent evals, but its advantage depends on a teacher reliably locating that boundary; the supplied results do not quantify this sensitivity or added cost.

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon ReasoningSkill²-Bench identifies capability transitions as a source of long-horizon failure, while Cliff offers a complementary training signal for locating the first step where such a trajectory becomes invalid.AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and VerificationFrontier models’ weakness at rejecting invalid proofs makes Cliff’s unreported sensitivity to teacher error consequential, because a mistaken boundary would assign token-level advantages incorrectly.RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward ConstructionRRC and Cliff address different reward bottlenecks: RRC preserves a judge’s relative rankings, whereas Cliff converts a detected first-error boundary into prefix-level advantages.Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and ReproducibilityThe reproducibility framework implies that Cliff’s reported gains require the full rollout, teacher, sampling, and compute protocol before they can be compared reliably with other training methods.

Context Map

model#reasoning#agent-evals

Uncertainty

Across **12 scenarios**, Cliff reportedly beat on-policy distillation by **15%** and standard GRPO by **7%**. The abstract does not specify task composition, compute overhead, or sensitivity to teacher misclassification.