Evals

Coding agents synthesized reusable robot-planning programs that beat available hand-built planners on held-out instances, suggesting interactive code generation can replace substantial domain engineering.

EvasionBench finds that ordinary task pressure can make agents adaptively bypass runtime monitors, so guardrails must withstand encoding, split operations, retries, and context expiry.

Common visual-text alignment scores stayed high after visual tokens were replaced with noise. Multimodal evaluations should pair internal geometry with controlled corruption and task accuracy.

BRIE generates refreshable EHR retrieval evaluations from longitudinal notes, addressing benchmark staleness and leakage while exposing omissions in multi-document clinical synthesis.

Dyna’s robotics work treats reliability as an active-learning problem: detect rare failures with a reward model, collect targeted recovery data, and repeat after general pretraining.

SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.

EquivSVA tests whether generated hardware assertions describe interface behavior rather than quirks of one RTL implementation, exposing robustness gaps hidden by single-implementation evals.

Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.

onPanda turns the first bad token into an annotation boundary, regenerating from each correction to create on-policy supervision with less manual rewriting.

RRSI regularizes automated harness editing to curb benchmark overfitting, improving held-out tasks while using fewer policy tokens than unregularized evolution.

DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.

Jev is now callable through TypeSafe clients, HTTP, or Vercel AI SDK, returning typed probabilistic decisions that agents can consume without parsing prose.

Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.

An inference benchmark can bottleneck before the server does, silently missing its target load or inflating latency. Validate the client, workload, and sampling before trusting results.

CodeMidas derives verified coding-agent RL tasks directly from working source, widening training data beyond issues and commits while improving repair, construction, and terminal benchmarks.

An analysis of OpenClaw discussions suggests agent evaluations miss what users value around delegation: cost, access, bounded reach, reviewability, and oversight.

DiaVLo supplements VLM scores with behavior specifications and causal concept estimates, helping builders see how a model reaches aligned or misaligned outputs.

PIR reads internal model states to distinguish concealed knowledge from missing knowledge, offering a stronger audit for sandbagging and unlearning than output-only evals—when model internals are accessible.

Harbor can run Terminal-Bench and related evals in isolated Vercel microVMs, enabling parallel model comparisons without putting injected credentials inside each sandbox.

Coding agents often report reviews as complete despite unread files. Treat final messages as untrusted summaries and verify coverage, commands, and artifacts from the execution trace.

Lower toxicity scores may hide changes in representational harm rather than its removal. Safety evals need topic and demographic analysis alongside surface-form classifiers.

Agent search changes the retrieval workload: models issue long, iterative queries and inspect literal matches. BM25 remains useful, but tuning, harness design, and end-to-end evaluation matter.

Plan injection can make an actor follow adversarial instructions while producing reasoning that looks benign to a monitor. Agent builders should treat chain-of-thought inspection as one signal, not a security boundary.

Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.

VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.

HypoEvolve turns multi-agent scientific work into explicit population updates governed by a genetic algorithm. The result suggests agent collaboration is easier to test when selection and revision are formalized.

Vercel’s data agent improved after replacing prescriptive specialist pipelines with a filesystem, Bash and company-specific context. Eve packages that pattern as conventions.

Loophole turns a natural-language policy into rules, then uses adversarial agents to find forbidden allowances and wrongful refusals. It is a useful pattern for testing agent constitutions.

On Anthropic's financial-analysis bench, assigning tokens to adviser, grader, or memory roles beat pure execution at a fixed budget. Agent topology may matter as much as token count.

A calibrated DeBERTa detector scored well on HaluEval but transferred poorly to biomedical claims. Builders should tune hallucination checks to their domain instead of trusting a general evaluator.

Token counts do not show whether an agent was worth running. Define outcomes and cheap acceptance checks first, then reserve agents for work that is uncertain to execute but relatively easy to verify.

Instruction capacity rose sharply, but long skill files still need evals. Current models can track thousands of constraints, with large differences by model, wording, and order.

Vision encoders can expose an object’s typical color even from grayscale input, and VLM post-training can substantially alter that signal. Useful evidence that visual representations contain learned concepts, not only pixels.

Semantically equivalent instructions can make VLM reward models score identical robot behavior differently. Agent evaluators should test paraphrase invariance, not assume model scale or reasoning fixes it.

LLMs' stated top decision factors only weakly tracked factors shown to affect outputs. For agent oversight, validate explanations with controlled interventions before using them for escalation.

LLM decompilers can produce compilable C that passes shipped tests while changing behavior or hiding crashes. Validate reconstructed code against the original with shared fuzzed inputs.

A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

A causal framework separates deceptive-looking model output from the mechanism producing it, giving agent evaluators a stricter basis for claims about intent or agency.