Open Brain
arXivPaperNeeds Review

Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

An evaluation study found that trace-based judging and security red-teaming surfaced different failures in analytics agents, supporting separate functional and adversarial checks.

arXiv · Nov 10, 2026
Open Source Open MarkdownOpen JSON
Source Summary

The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.

Practical Implication

For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.

Agent-Ready Context
The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.

For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.

The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.
Connected Context · Feed7 Judgment

This comparison suggests evaluation methods catch different failures: trace-based judging performed better than text-only judging on the tested cases, while functional testing and security red-teaming found additional failures with no overlap. The evidence is limited to two analytics agents, so the relative performance may not transfer to other settings.

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware ScreenBoth warn that agent evaluation can mislead when detached from execution evidence: this study favors trace-based judging over text-only judging, while the repair study finds compile rate and similarity can reward broken security patches.Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon LabsThe study finds complementary failures from functional testing and security red-teaming; Vending-Bench likewise motivates combining repeatable simulations with real-world tests because behavior can change under evaluation conditions.Domain-Specific Hallucination Detection in Large Language ModelsBoth narrow confidence in a single evaluator: text-only judging generated substantially more false alarms in this study, while the hallucination detector's transfer drop shows that evaluator performance depends on domain and context.
Context Map
benchmarkdatasecurity#agent-evals#agent-reliability#benchmark-integrity
Uncertainty
The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.