{
  "schema_version": "1.0",
  "id": "archive:https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
  "slug": "demystifying-evals-for-ai-agents-1kh2tdz",
  "url": "https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz",
  "title": "Demystifying evals for AI agents",
  "why_included": "Anthropic's practical guide to agent evals: grader types, pass@k vs pass^k, and a start-small roadmap (20-50 tasks from real failures). Teams with evals adopt new models in days instead of weeks.",
  "summary": "**The gist** Anthropic published a practical guide to evaluating AI agents. It defines core terms (**pass@k vs pass^k**, trials, transcripts, graders), compares **code-based, model-based, and human graders**, and walks through eval patterns for coding, conversational, research, and computer-use agents, citing benchmarks like **SWE-bench Verified** and **τ-Bench**.",
  "practical_implication": "**Why it matters** The roadmap is directly usable: start with **20-50 tasks** drawn from real failures, prefer deterministic graders, read transcripts to check your graders, and retire saturated evals into regression suites. Anthropic claims teams with evals adopt new models **in days instead of weeks** because regressions surface immediately.",
  "agent_context": "**The gist** Anthropic published a practical guide to evaluating AI agents. It defines core terms (**pass@k vs pass^k**, trials, transcripts, graders), compares **code-based, model-based, and human graders**, and walks through eval patterns for coding, conversational, research, and computer-use agents, citing benchmarks like **SWE-bench Verified** and **τ-Bench**.\n\n**Why it matters** The roadmap is directly usable: start with **20-50 tasks** drawn from real failures, prefer deterministic graders, read transcripts to check your graders, and retire saturated evals into regression suites. Anthropic claims teams with evals adopt new models **in days instead of weeks** because regressions surface immediately.\n\n**Watch out** Eval bugs distort results — a grading fix moved **Opus 4.5 from 42% to 95%** on CORE-Bench, and a **0% pass rate** usually means a broken task, not a weak agent. LLM judges need human calibration, and evals miss creative agent behavior, so pair them with production monitoring.",
  "source": {
    "name": "Anthropic",
    "url": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Engineering Post",
  "layer": "benchmark",
  "domains": [],
  "topics": [
    "agent-evals",
    "benchmark-integrity",
    "agent-reliability"
  ],
  "verification": {
    "status": "official_source",
    "label": "Official Source",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "Eval bugs distort results — a grading fix moved **Opus 4.5 from 42% to 95%** on CORE-Bench, and a **0% pass rate** usually means a broken task, not a weak agent. LLM judges need human calibration, and evals miss creative agent behavior, so pair them with production monitoring."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz",
    "json": "https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz.json",
    "markdown": "https://feed7.dev/p/demystifying-evals-for-ai-agents-1kh2tdz.md"
  }
}