Benchmark Integrity
Current Answer
No editorial synthesis yet — the evidence below is collected automatically from source labels. A current answer lands here once an editor approves one.
Evidence
Terminal-Bench 2.1 fixes issues found in roughly 30% of its tasks, shifting accuracy and leaderboard positions. The audit also shows that time limits can change apparent harness performance.
CFR tests whether animal identification models rely on the animal or on background cues by comparing retrieval across full-image, foreground-only, and shortcut-only views.
CATT isolates factors that affect language-model data attribution. The paper finds that focusing gradient scores on key tokens improves recall over document-wide averaging.
STRATA grades the evidence and statistical choices behind temporal claims, not just the final verdict. Its experiments find that models often fail when selecting which records to use.
A study of six point-in-time models finds that knowledge with more variable representation across web years often improves faster later in pretraining. Its temporal variability score offers a diagnostic beyond aggregate benchmark results.
A study of Jev-style models finds that repeatable answers can still become wrong when option order changes. Stable accuracy alone does not show that a model resists selection bias.
Common visual-text alignment scores stayed high after visual tokens were replaced with noise. Multimodal evaluations should pair internal geometry with controlled corruption and task accuracy.
BRIE generates refreshable EHR retrieval evaluations from longitudinal notes, addressing benchmark staleness and leakage while exposing omissions in multi-document clinical synthesis.
SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.
EquivSVA tests whether generated hardware assertions describe interface behavior rather than quirks of one RTL implementation, exposing robustness gaps hidden by single-implementation evals.
Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.
DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.
Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.
An inference benchmark can bottleneck before the server does, silently missing its target load or inflating latency. Validate the client, workload, and sampling before trusting results.
DiaVLo supplements VLM scores with behavior specifications and causal concept estimates, helping builders see how a model reaches aligned or misaligned outputs.
PIR reads internal model states to distinguish concealed knowledge from missing knowledge, offering a stronger audit for sandbagging and unlearning than output-only evals—when model internals are accessible.
Lower toxicity scores may hide changes in representational harm rather than its removal. Safety evals need topic and demographic analysis alongside surface-form classifiers.
Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.
VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.
For edge vision, specialized training outweighed model size: 300M-parameter BioCLIP beat 2–8B VLMs, while field imagery degraded every model and open-set prompts produced invented species.
A calibrated DeBERTa detector scored well on HaluEval but transferred poorly to biomedical claims. Builders should tune hallucination checks to their domain instead of trusting a general evaluator.
Visual embedding initialization gave a small language model durable object-property knowledge that most standard benchmarks missed, showing how broad eval suites can hide narrow causal gains.
HuBERT and wav2vec 2.0 appear to encode word identity beyond local phonetics in later layers. The paper offers a cleaner probe for builders evaluating speech representations.
Semantically equivalent instructions can make VLM reward models score identical robot behavior differently. Agent evaluators should test paraphrase invariance, not assume model scale or reasoning fixes it.
Molecular benchmark accuracy can reflect retrieval of published values rather than prediction. Contamination audits should test digit-level recall and repeat runs across reasoning settings.
LLM decompilers can produce compilable C that passes shipped tests while changing behavior or hiding crashes. Validate reconstructed code against the original with shared fuzzed inputs.
A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.
Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.
SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.
A causal framework separates deceptive-looking model output from the mechanism producing it, giving agent evaluators a stricter basis for claims about intent or agency.
User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.
Mechanistic tests suggest two open LLM judges route local errors early and form ratings late, giving eval builders concrete layers and components to probe for judge failures.
BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.
DeepMind’s panel shows why generative-media evals need task-specific human review: broad preferences can miss repeated artifacts, exact sizing, text errors, and brand consistency.
SWE-Prime finds that filtering coding-agent traces by process and segment quality can beat training on every resolved trajectory, reducing noisy imitation from redundant or risky steps.
A multi-GPU kernel benchmark shows coding agents can improve with tools and iteration, but still struggle with communication scheduling, partitioning, and transfer choices.
Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.
Einstein Arena suggests multi-agent environments can outperform fixed workflows when they expose verifiers, shared solutions, forums, and incentives. Its results also show why benchmark shortcuts need active testing.
ConceptGuard tests whether model unlearning blocks harmful uses of a concept while preserving benign ones. Current methods show weak contextual control and sharp forgetting-versus-utility trade-offs.
Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.