# Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Source: [AI Engineer](https://www.youtube.com/watch?v=jRCpXUjz4CI)  
Feed7 permalink: https://feed7.dev/p/everything-is-a-rollout-alex-shaw-ryan-marten-terminal-bench-harbor-laud-0iz4rgx  
Published: 2026-07-24T16:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Harbor frames agent development as an empirical loop: run agents in reproducible sandboxes, verify outcomes, inspect trajectories, and evaluate every harness or model change.

## Source Summary

**Harbor** specifies agent environments and runs any agent, model, sandbox, and task combination in parallel. Its registry reportedly contains **three to four hundred eval sets**, with support for separate verification sandboxes, artifacts, and simulated users.

## Practical Implication

Treat prompts, skills, tools, and model choices like tunable system parameters. Run repeated rollouts, grade outcomes, inspect recurring failures, and require measured improvement before merging harness changes.

## Agent-Ready Context

**Harbor** specifies agent environments and runs any agent, model, sandbox, and task combination in parallel. Its registry reportedly contains **three to four hundred eval sets**, with support for separate verification sandboxes, artifacts, and simulated users.

Treat prompts, skills, tools, and model choices like tunable system parameters. Run repeated rollouts, grade outcomes, inspect recurring failures, and require measured improvement before merging harness changes.

A rollout framework supplies infrastructure, not a useful eval by itself. Builders still need representative tasks, reliable verifiers, and checks for reward hacking or overfitting to the evaluation set.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, sandboxing

## Uncertainty

- A rollout framework supplies infrastructure, not a useful eval by itself. Builders still need representative tasks, reliable verifiers, and checks for reward hacking or overfitting to the evaluation set.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.