Open Brain
VercelEngineering PostOfficial Source

Agents are users, so your software needs evals

For agent tasks, repeated end-to-end evals catch wrong final outcomes that logs and one-off checks can miss; choose pass@k or pass^k based on the cost of failure.

Vercel · Nov 16, 2026
Open Source Open MarkdownOpen JSON
Source Summary

The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss. Pick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure. The article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.

Practical Implication

No separate implication has been recorded yet.

Agent-Ready Context
The article groups agent tests into **deterministic tests (0 LLM calls)**, **one-turn evals (1)**, and **end-to-end evals (2+)**. E2E trials run a task in an environment and check its final result; repeated trials reveal variation that a single manual run or ordinary logs may miss.
Pick the lowest-cost test tier that catches the failure you care about. For coding agents, keep deterministic checks for predictable behavior, use one-turn evals for individual decisions, and run repeated E2E tasks when the whole sequence determines whether the user’s goal was met. Track success rate, efficiency, and pass@k or pass^k according to the cost of failure.
The article’s example shows why those last two measures differ: 60% success across trials yields roughly 99% pass@5 but only 8% pass^5. E2E coverage of every possible agent behavior is impractical, so start with a narrow user outcome and a handful of tasks.
Connected Context · Feed7 Judgment

This article gives a practical way to match evaluation cost to the failure being measured: deterministic checks for predictable behavior, one-turn evals for individual choices, and repeated end-to-end trials for whole-task outcomes. Its pass@k versus pass^k example shows why occasional success differs from reliable success. Start with a small set of user outcomes rather than trying to cover every behavior.

Context Map
benchmarkcoding#agent-evals#agent-reliability#benchmark-integrity