Benchmark Integrity

Common visual-text alignment scores stayed high after visual tokens were replaced with noise. Multimodal evaluations should pair internal geometry with controlled corruption and task accuracy.

BRIE generates refreshable EHR retrieval evaluations from longitudinal notes, addressing benchmark staleness and leakage while exposing omissions in multi-document clinical synthesis.

SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.

EquivSVA tests whether generated hardware assertions describe interface behavior rather than quirks of one RTL implementation, exposing robustness gaps hidden by single-implementation evals.

Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.

DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.

Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.

An inference benchmark can bottleneck before the server does, silently missing its target load or inflating latency. Validate the client, workload, and sampling before trusting results.

DiaVLo supplements VLM scores with behavior specifications and causal concept estimates, helping builders see how a model reaches aligned or misaligned outputs.

PIR reads internal model states to distinguish concealed knowledge from missing knowledge, offering a stronger audit for sandbagging and unlearning than output-only evals—when model internals are accessible.

Lower toxicity scores may hide changes in representational harm rather than its removal. Safety evals need topic and demographic analysis alongside surface-form classifiers.

Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.

VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.

For edge vision, specialized training outweighed model size: 300M-parameter BioCLIP beat 2–8B VLMs, while field imagery degraded every model and open-set prompts produced invented species.

A calibrated DeBERTa detector scored well on HaluEval but transferred poorly to biomedical claims. Builders should tune hallucination checks to their domain instead of trusting a general evaluator.

Visual embedding initialization gave a small language model durable object-property knowledge that most standard benchmarks missed, showing how broad eval suites can hide narrow causal gains.

HuBERT and wav2vec 2.0 appear to encode word identity beyond local phonetics in later layers. The paper offers a cleaner probe for builders evaluating speech representations.

Semantically equivalent instructions can make VLM reward models score identical robot behavior differently. Agent evaluators should test paraphrase invariance, not assume model scale or reasoning fixes it.

Molecular benchmark accuracy can reflect retrieval of published values rather than prediction. Contamination audits should test digit-level recall and repeat runs across reasoning settings.

LLM decompilers can produce compilable C that passes shipped tests while changing behavior or hiding crashes. Validate reconstructed code against the original with shared fuzzed inputs.

A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

A causal framework separates deceptive-looking model output from the mechanism producing it, giving agent evaluators a stricter basis for claims about intent or agency.

User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.

Mechanistic tests suggest two open LLM judges route local errors early and form ratings late, giving eval builders concrete layers and components to probe for judge failures.

BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.

DeepMind’s panel shows why generative-media evals need task-specific human review: broad preferences can miss repeated artifacts, exact sizing, text errors, and brand consistency.

SWE-Prime finds that filtering coding-agent traces by process and segment quality can beat training on every resolved trajectory, reducing noisy imitation from redundant or risky steps.

A multi-GPU kernel benchmark shows coding agents can improve with tools and iteration, but still struggle with communication scheduling, partitioning, and transfer choices.

Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.

Einstein Arena suggests multi-agent environments can outperform fixed workflows when they expose verifiers, shared solutions, forums, and incentives. Its results also show why benchmark shortcuts need active testing.

ConceptGuard tests whether model unlearning blocks harmful uses of a concept while preserving benign ones. Current methods show weak contextual control and sharp forgetting-versus-utility trade-offs.

Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.

TokEval links tokenizer properties to language, math, and code performance, offering cheaper screening signals before committing compute to pretraining sweeps.

A controlled study encoded authenticated internal-state evidence into unchanged answers, suggesting generated text could carry provenance signals, but not that current models reveal them naturally.

Tested compliance guards often ignored the governing rule and classified from scenario cues. Builders should counterfactually swap policies before trusting a detector as an audit control.

Static computer-use benchmarks can reward memorized action scripts rather than adaptation. Vary task state, verify every generated case, and calculate uncertainty across both actions and environments.

QuoteBench shows that shell-command scores can conceal failures introduced by serialization and reparsing. Agent evals should identify the execution path, not attribute every result to the model.

LittleLearner offers a controlled model and corpus for studying knowledge acquisition without unknown prior exposure. Its initial results separate better use of known material from new capability.