arXivPaperNeeds Review

LLM-as-a-Verifier: A General-Purpose Verification Framework

Training-free framework that turns an LLM judge's token logits into continuous scores for verifying agent outputs — 78.2% on SWE-Bench Verified, 86.5% on Terminal-Bench V2 — and ships a Claude Code extension.

arXiv

Open Source Open Markdown Open JSON

Source Summary

**The gist** LLM-as-a-Verifier scores candidate solutions by taking the **expectation over scoring-token logits**, producing continuous rather than discrete grades with no extra training. Scaled via granularity, repeated evaluation, and criteria decomposition, it reaches **86.5% on Terminal-Bench V2**, **78.2% on SWE-Bench Verified**, and **87.4% on RoboRewardBench**, with a cost-aware algorithm for ranking candidates.

Practical Implication

**Why it matters** A training-free verifier is the piece that makes best-of-N agent runs and self-checking loops pay off. The authors ship a **Claude Code extension** that turns the scores into a live task-progress signal for monitoring agentic systems, and the same dense feedback improves RL sample efficiency for **SAC and GRPO**.

Agent-Ready Context

**The gist** LLM-as-a-Verifier scores candidate solutions by taking the **expectation over scoring-token logits**, producing continuous rather than discrete grades with no extra training. Scaled via granularity, repeated evaluation, and criteria decomposition, it reaches **86.5% on Terminal-Bench V2**, **78.2% on SWE-Bench Verified**, and **87.4% on RoboRewardBench**, with a cost-aware algorithm for ranking candidates.

**Why it matters** A training-free verifier is the piece that makes best-of-N agent runs and self-checking loops pay off. The authors ship a **Claude Code extension** that turns the scores into a live task-progress signal for monitoring agentic systems, and the same dense feedback improves RL sample efficiency for **SAC and GRPO**.

**Watch out** The method needs **logprob access** to scoring tokens, which not every hosted API exposes, and its gains come from spending more judge calls — repeated evaluation and criteria decomposition multiply **verification cost**. Judge-side bias isn't addressed in the material.

Context Map

benchmarkcodingresearch#agent-evals#agent-reliability#harness-engineering

Uncertainty

The method needs **logprob access** to scoring tokens, which not every hosted API exposes, and its gains come from spending more judge calls — repeated evaluation and criteria decomposition multiply **verification cost**. Judge-side bias isn't addressed in the material.