# Demystifying evals for AI agents

Source: [Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)  
Feed7 permalink: https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz  
Published: Unknown  
Trust: Official Source (official_source)

## Why Included

Anthropic's practical guide to agent evals: grader types, pass@k vs pass^k, and a start-small roadmap (20-50 tasks from real failures). Teams with evals adopt new models in days instead of weeks.

## Source Summary

**The gist** Anthropic published a practical guide to evaluating AI agents. It defines core terms (**pass@k vs pass^k**, trials, transcripts, graders), compares **code-based, model-based, and human graders**, and walks through eval patterns for coding, conversational, research, and computer-use agents, citing benchmarks like **SWE-bench Verified** and **τ-Bench**.

## Practical Implication

**Why it matters** The roadmap is directly usable: start with **20-50 tasks** drawn from real failures, prefer deterministic graders, read transcripts to check your graders, and retire saturated evals into regression suites. Anthropic claims teams with evals adopt new models **in days instead of weeks** because regressions surface immediately.

## Agent-Ready Context

**The gist** Anthropic published a practical guide to evaluating AI agents. It defines core terms (**pass@k vs pass^k**, trials, transcripts, graders), compares **code-based, model-based, and human graders**, and walks through eval patterns for coding, conversational, research, and computer-use agents, citing benchmarks like **SWE-bench Verified** and **τ-Bench**.

**Why it matters** The roadmap is directly usable: start with **20-50 tasks** drawn from real failures, prefer deterministic graders, read transcripts to check your graders, and retire saturated evals into regression suites. Anthropic claims teams with evals adopt new models **in days instead of weeks** because regressions surface immediately.

**Watch out** Eval bugs distort results — a grading fix moved **Opus 4.5 from 42% to 95%** on CORE-Bench, and a **0% pass rate** usually means a broken task, not a weak agent. LLM judges need human calibration, and evals miss creative agent behavior, so pair them with production monitoring.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- Eval bugs distort results — a grading fix moved **Opus 4.5 from 42% to 95%** on CORE-Bench, and a **0% pass rate** usually means a broken task, not a weak agent. LLM judges need human calibration, and evals miss creative agent behavior, so pair them with production monitoring.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.