Open Brain
AI EngineerVideoSource Linked

How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

Warp’s Zach Lloyd argues that teams can govern and improve coding agents by running them in sandboxed software factories, with workflows and evaluation defined in code. He recommends testing models on real team tasks.

AI Engineer · Nov 9, 2026
Open Source Open MarkdownOpen JSON
Source Summary

Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness.

Practical Implication

If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks.

Agent-Ready Context
Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness.

If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks.

The talk presents Warp Factories as one implementation, but the factory approach is broader. Evaluation can use an LLM judge, which Lloyd says may be supplemented by human grading; the examples do not establish that this approach fits every team or task.
Connected Context · Feed7 Judgment

Lloyd’s factory proposal connects centralized, versioned workflows with replayed, task-specific evaluation for routing and improvement. The candidates sharpen its limits and complements: production traces can seed evaluations, scores depend on harnesses, and stronger workflow automation still needs human ownership and task-relevant evidence.

Agents That Self Improve While You Sleep - Fuad Ali, Arize AIBoth use recorded runs to improve agent workflows; this candidate specifies a trace-to-fix loop that promotes recurring failures into regression evaluations with human review.Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChainIts trace-mining approach supplies another source for the factory’s representative tasks and evaluations, while explicitly flagging misleading proxies and trace-scale costs.Designing Agents (The Floor Is the Frontier) — Ben Hylak, RaindropThis candidate narrows how factory benchmarks should be read: harness-coupled evaluations can decay, so production failures and explicit metrics should anchor checks.Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayerThe factory’s throughput and automation proposal is qualified by this candidate’s emphasis on maintainability, upfront alignment, and continued human ownership of code.
Context Map
agentcoding#harness-engineering#cloud-agents#agent-evals
Uncertainty
The talk presents Warp Factories as one implementation, but the factory approach is broader. Evaluation can use an LLM judge, which Lloyd says may be supplemented by human grading; the examples do not establish that this approach fits every team or task.