# Agents are users, so your software needs evals

Source: [Vercel](https://vercel.com/blog/agent-evals)  
Feed7 permalink: https://feed7.dev/p/agent-evals-13euax7  
Published: 2026-11-16T05:00:00.000Z  
Trust: Official Source (official_source)

## Why Included

For agent tasks, repeated end-to-end evals catch wrong final outcomes that logs and one-off checks can miss; choose pass@k or pass^k based on the cost of failure.

## Source Summary

The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss.
Pick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure.
The article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.

## Practical Implication

No separate implication recorded.

## Agent-Ready Context

The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss.
Pick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure.
The article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.

## Connected Context

Feed7 judgment across 1116 accumulated Signals:

This article gives a practical way to match evaluation cost to the failure being measured: deterministic checks for predictable behavior, one-turn evals for individual choices, and repeated end-to-end trials for whole-task outcomes. Its pass@k versus pass^k example shows why occasional success differs from reliable success. Start with a small set of user outcomes rather than trying to cover every behavior.

- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — The article’s layered testing approach can measure task outcomes, while SWE-Gate shows that functional success alone can miss review-derived constraints; those need a separate acceptance check.
- [Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software](https://feed7.dev/p/rethinking-environments-for-long-horizon-work-rayan-garg-theta-software-11r7wbx) — Both emphasize judging end-to-end work by its result; the long-horizon environments talk adds that final state and queryable trajectories matter when tasks permit different paths.
- [Demystifying evals for AI agents](https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz) — Anthropic’s guide independently supports starting with real failures and distinguishing pass@k from pass^k, reinforcing this article’s small, outcome-focused evaluation plan.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, benchmark-integrity

## Uncertainty

- None recorded.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
