Benchmark Integrity

User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.

Mechanistic tests suggest two open LLM judges route local errors early and form ratings late, giving eval builders concrete layers and components to probe for judge failures.

BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.

DeepMind’s panel shows why generative-media evals need task-specific human review: broad preferences can miss repeated artifacts, exact sizing, text errors, and brand consistency.

SWE-Prime finds that filtering coding-agent traces by process and segment quality can beat training on every resolved trajectory, reducing noisy imitation from redundant or risky steps.

A multi-GPU kernel benchmark shows coding agents can improve with tools and iteration, but still struggle with communication scheduling, partitioning, and transfer choices.

Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.

Einstein Arena suggests multi-agent environments can outperform fixed workflows when they expose verifiers, shared solutions, forums, and incentives. Its results also show why benchmark shortcuts need active testing.

ConceptGuard tests whether model unlearning blocks harmful uses of a concept while preserving benign ones. Current methods show weak contextual control and sharp forgetting-versus-utility trade-offs.

Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.

TokEval links tokenizer properties to language, math, and code performance, offering cheaper screening signals before committing compute to pretraining sweeps.

A controlled study encoded authenticated internal-state evidence into unchanged answers, suggesting generated text could carry provenance signals, but not that current models reveal them naturally.

Tested compliance guards often ignored the governing rule and classified from scenario cues. Builders should counterfactually swap policies before trusting a detector as an audit control.

Static computer-use benchmarks can reward memorized action scripts rather than adaptation. Vary task state, verify every generated case, and calculate uncertainty across both actions and environments.

QuoteBench shows that shell-command scores can conceal failures introduced by serialization and reparsing. Agent evals should identify the execution path, not attribute every result to the model.

LittleLearner offers a controlled model and corpus for studying knowledge acquisition without unknown prior exposure. Its initial results separate better use of known material from new capability.

A task-agnostic influence measure tracks which examples steer pretraining toward final parameters without choosing a downstream eval. It reveals a literature-to-STEM shift across training.

Cursor’s AIUC-1 certification combines a controls audit with adversarial testing of live agents. For enterprise evaluation, it adds behavioral evidence beyond conventional data-security attestations.

P-Bench tests whether agents choose statistically valid methods, not merely execute code. Fisher-R1-14B improved single-trial success over DeepSeek-V4-Pro by 21% on average.

SABRE turns a Markdown test design into generated VLM stress tests, then filters and repairs candidates. It offers a repeatable pattern for refreshing evals as models improve.

Controlled traces show video models can improve final counting scores without faithfully recovering events, so agents handling video need timestamp-level checks, not answer-only evals.

SocietyBench tests forecasting in anonymized social timelines, exposing gaps that task-completion evals miss and showing agent frameworks did not improve the shared base model.

A live, pre-kickoff benchmark removes answer leakage by construction and finds six frontier models clustered near a bookmaker-favorite baseline, with no gain from majority voting.

This paper separates three test-time scaling regimes and argues that model evals must report the full inference protocol, compute accounting, uncertainty, and replay artifacts.

onepot-Bench 0 evaluates chemistry models with three lab-oriented tests, including private experimental data to reduce contamination risk. It is a useful eval design pattern, though results and reproducibility are absent here.

Nick Heiner argues that leaderboard gains can diverge from useful agent behavior through contamination, weak verifiers, reward hacking, and test conditions that users cannot inspect.

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

Biology agents need evaluators that verify analysis of large experimental datasets, not recall. LatchBio found human review essential because valid scientific paths can defeat brittle graders.

FriendBench shows why aggregate accuracy can hide behavioral bias: top multimodal models matched human panels overall but favored the “stranger” answer and gained less from video.

Coding benchmark scores can mislead when prompts leak implementation details or tests reject valid solutions. Inspect tasks and verifiers before using a leaderboard to choose an agent.

Change2Task converts merged PR history into executable coding-agent tasks on current repository states. It could expand continuous eval sets while reducing stale environments and repeated setup.

PAIChecker finds PR–issue mismatch in 13.6% of SWE-bench Verified instances. Benchmark maintainers should validate that the issue, patch, and tests describe the same task before trusting agent scores.

At equal generated-token cost, repeated sampling matched or beat seven reflection, critique, selection, and debate methods. Agent evals should budget every generated token, not compare against one-shot baselines.

Synthetic personas can extend existing research, but they are forecasts, not extra respondents. Ground prompts richly and validate each setup against human data before using it.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

A 1,215-run study finds entity-matching architecture and model variant matter more than scale alone; generative matchers mainly help under distribution shift.

A delayed KV-eviction policy helps when memory reuse is sharp and observable, but loses its edge on natural-text benchmarks—a useful warning for agent-memory claims.

DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.

Real workflow traces may teach agents more than manufactured tasks, while benchmark scores can shift with the harness. Build pipelines around live work and test across scaffolds.