How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp
Warp’s Zach Lloyd argues that teams can govern and improve coding agents by running them in sandboxed software factories, with workflows and evaluation defined in code. He recommends testing models on real team tasks.
Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness.
If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks.
Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness. If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks. The talk presents Warp Factories as one implementation, but the factory approach is broader. Evaluation can use an LLM judge, which Lloyd says may be supplemented by human grading; the examples do not establish that this approach fits every team or task.
Lloyd’s factory proposal connects centralized, versioned workflows with replayed, task-specific evaluation for routing and improvement. The candidates sharpen its limits and complements: production traces can seed evaluations, scores depend on harnesses, and stronger workflow automation still needs human ownership and task-relevant evidence.