Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
An evaluation study found that trace-based judging and security red-teaming surfaced different failures in analytics agents, supporting separate functional and adversarial checks.
The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.
For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.
The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap. For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study. The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.
This comparison suggests evaluation methods catch different failures: trace-based judging performed better than text-only judging on the tested cases, while functional testing and security red-teaming found additional failures with no overlap. The evidence is limited to two analytics agents, so the relative performance may not transfer to other settings.