{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2611.14493v1",
  "slug": "2611-14493v1-1ghdgem",
  "url": "https://feed7.dev/p/2611-14493v1-1ghdgem",
  "title": "Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents",
  "why_included": "An evaluation study found that trace-based judging and security red-teaming surfaced different failures in analytics agents, supporting separate functional and adversarial checks.",
  "summary": "The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.",
  "practical_implication": "For builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.",
  "agent_context": "The study compared three evaluation approaches on two analytics agents using frozen traces. On **72 ground-truth cases**, trace-based judging reached **F1 0.90**; text-only judging matched its recall but produced **5× more false alarms**. Functional testing and security red-teaming found 24 new failures with no overlap.\n\nFor builders running multi-tool agents, preserve execution traces for functional evaluation and add adversarial red-teaming for security. A single text-only judge may flag more cases without improving recall, while deterministic heuristics represent a third evaluation approach in the study.\n\nThe results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2611.14493v1",
    "published_at": "2026-11-10T18:50:06.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "data",
    "security"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability",
    "benchmark-integrity"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The results come from two analytics agents in a controlled research setting. The paper does not show whether the same performance differences hold across other agents or tasks."
  ],
  "connected_context": {
    "meaning": "This comparison suggests evaluation methods catch different failures: trace-based judging performed better than text-only judging on the tested cases, while functional testing and security red-teaming found additional failures with no overlap. The evidence is limited to two analytics agents, so the relative performance may not transfer to other settings.",
    "corpus_size": 1059,
    "generated_at": "2026-11-11T11:09:28.413Z",
    "connections": [
      {
        "title": "Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.26749v1",
        "feed7_url": "https://feed7.dev/p/2609-26749v1-1fj6jga",
        "reason": "Both warn that agent evaluation can mislead when detached from execution evidence: this study favors trace-based judging over text-only judging, while the repair study finds compile rate and similarity can reward broken security patches."
      },
      {
        "title": "Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=cO8qC6HBuBg",
        "feed7_url": "https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz",
        "reason": "The study finds complementary failures from functional testing and security red-teaming; Vending-Bench likewise motivates combining repeatable simulations with real-world tests because behavior can change under evaluation conditions."
      },
      {
        "title": "Domain-Specific Hallucination Detection in Large Language Models",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.11878v1",
        "feed7_url": "https://feed7.dev/p/2609-11878v1-1ltrudx",
        "reason": "Both narrow confidence in a single evaluator: text-only judging generated substantially more false alarms in this study, while the hallucination detector's transfer drop shows that evaluator performance depends on domain and context."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-11-10T18:50:06.000Z",
  "modified_at": "2026-11-10T18:50:06.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2611-14493v1-1ghdgem",
    "json": "https://feed7.dev/p/2611-14493v1-1ghdgem.json",
    "markdown": "https://feed7.dev/p/2611-14493v1-1ghdgem.md"
  }
}