{
  "schema_version": "1.1",
  "id": "archive:https://vercel.com/blog/agent-evals",
  "slug": "agent-evals-13euax7",
  "url": "https://feed7.dev/p/agent-evals-13euax7",
  "title": "Agents are users, so your software needs evals",
  "why_included": "For agent tasks, repeated end-to-end evals catch wrong final outcomes that logs and one-off checks can miss; choose pass@k or pass^k based on the cost of failure.",
  "summary": "The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss.\nPick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure.\nThe article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.",
  "practical_implication": "",
  "agent_context": "The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss.\nPick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure.\nThe article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.",
  "source": {
    "name": "Vercel",
    "url": "https://vercel.com/blog/agent-evals",
    "published_at": "2026-11-16T05:00:00.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Engineering Post",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability",
    "benchmark-integrity"
  ],
  "verification": {
    "status": "official_source",
    "label": "Official Source",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [],
  "connected_context": {
    "meaning": "This article gives a practical way to match evaluation cost to the failure being measured: deterministic checks for predictable behavior, one-turn evals for individual choices, and repeated end-to-end trials for whole-task outcomes. Its pass@k versus pass^k example shows why occasional success differs from reliable success. Start with a small set of user outcomes rather than trying to cover every behavior.",
    "corpus_size": 1116,
    "generated_at": "2026-11-18T11:08:28.288Z",
    "connections": [
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "The article’s layered testing approach can measure task outcomes, while SWE-Gate shows that functional success alone can miss review-derived constraints; those need a separate acceptance check."
      },
      {
        "title": "Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=2aS7aKoXn64",
        "feed7_url": "https://feed7.dev/p/rethinking-environments-for-long-horizon-work-rayan-garg-theta-software-11r7wbx",
        "reason": "Both emphasize judging end-to-end work by its result; the long-horizon environments talk adds that final state and queryable trajectories matter when tasks permit different paths."
      },
      {
        "title": "Demystifying evals for AI agents",
        "source_name": "Anthropic",
        "source_url": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
        "feed7_url": "https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz",
        "reason": "Anthropic’s guide independently supports starting with real failures and distinguishing pass@k from pass^k, reinforcing this article’s small, outcome-focused evaluation plan."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-11-16T05:00:00.000Z",
  "modified_at": "2026-11-16T05:00:00.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/agent-evals-13euax7",
    "json": "https://feed7.dev/p/agent-evals-13euax7.json",
    "markdown": "https://feed7.dev/p/agent-evals-13euax7.md"
  }
}