# Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

Source: [arXiv](https://arxiv.org/abs/2611.14493v1)  
Feed7 permalink: https://feed7.dev/p/2611-14493v1-1ghdgem  
Published: 2026-11-10T18:50:06.000Z  
Trust: Needs Review (needs_review)

## Why Included

An evaluation study found that trace-based judging and security red-teaming surfaced different failures in analytics agents, supporting separate functional and adversarial checks.

## Source Summary

The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.

## Practical Implication

For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.

## Agent-Ready Context

The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.

For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.

The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.

## Connected Context

Feed7 judgment across 1059 accumulated Signals:

This comparison suggests evaluation methods catch different failures: trace-based judging performed better than text-only judging on the tested cases, while functional testing and security red-teaming found additional failures with no overlap. The evidence is limited to two analytics agents, so the relative performance may not transfer to other settings.

- [Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen](https://feed7.dev/p/2609-26749v1-1fj6jga) — Both warn that agent evaluation can mislead when detached from execution evidence: this study favors trace-based judging over text-only judging, while the repair study finds compile rate and similarity can reward broken security patches.
- [Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs](https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz) — The study finds complementary failures from functional testing and security red-teaming; Vending-Bench likewise motivates combining repeatable simulations with real-world tests because behavior can change under evaluation conditions.
- [Domain-Specific Hallucination Detection in Large Language Models](https://feed7.dev/p/2609-11878v1-1ltrudx) — Both narrow confidence in a single evaluator: text-only judging generated substantially more false alarms in this study, while the hallucination detector's transfer drop shows that evaluator performance depends on domain and context.

## Context Map

- Layer: benchmark
- Domains: data, security
- Topics: agent-evals, agent-reliability, benchmark-integrity

## Uncertainty

- The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
