# How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

Source: [AI Engineer](https://www.youtube.com/watch?v=nKjPBAw_N_0)  
Feed7 permalink: https://feed7.dev/p/how-to-build-a-software-factory-that-improves-itself-zach-lloyd-warp-050811m  
Published: 2026-11-09T08:00:17.000Z  
Trust: Source Linked (source_linked)

## Why Included

Warp’s Zach Lloyd argues that teams can govern and improve coding agents by running them in sandboxed software factories, with workflows and evaluation defined in code. He recommends testing models on real team tasks.

## Source Summary

Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness.

## Practical Implication

If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks.

## Agent-Ready Context

Lloyd describes a shift from engineers prompting agents on laptops to **sandboxed cloud factories**. He proposes keeping workflows, agent definitions, skills, MCPs, and benchmarks in a versioned repository, then using recorded runs and task-specific evaluations to improve the system. One example compares models on replayed coding tasks by cost and judged correctness.

If you run agents across a team, define the workflow and controls centrally, capture past tasks, and build a benchmark from representative work. Use those results to guide model routing and factory changes instead of relying only on public benchmarks.

The talk presents Warp Factories as one implementation, but the factory approach is broader. Evaluation can use an LLM judge, which Lloyd says may be supplemented by human grading; the examples do not establish that this approach fits every team or task.

## Connected Context

Feed7 judgment across 1045 accumulated Signals:

Lloyd’s factory proposal connects centralized, versioned workflows with replayed, task-specific evaluation for routing and improvement. The candidates sharpen its limits and complements: production traces can seed evaluations, scores depend on harnesses, and stronger workflow automation still needs human ownership and task-relevant evidence.

- [Agents That Self Improve While You Sleep - Fuad Ali, Arize AI](https://feed7.dev/p/agents-that-self-improve-while-you-sleep-fuad-ali-arize-ai-19hmq9b) — Both use recorded runs to improve agent workflows; this candidate specifies a trace-to-fix loop that promotes recurring failures into regression evaluations with human review.
- [Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain](https://feed7.dev/p/improving-agents-is-a-data-mining-problem-vivek-trivedy-langchain-1cgibr8) — Its trace-mining approach supplies another source for the factory’s representative tasks and evaluations, while explicitly flagging misleading proxies and trace-scale costs.
- [Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop](https://feed7.dev/p/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-0uoems4) — This candidate narrows how factory benchmarks should be read: harness-coupled evaluations can decay, so production failures and explicit metrics should anchor checks.
- [Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer](https://feed7.dev/p/harness-engineering-is-not-enough-why-software-factories-fail-dex-horthy-1tj1amt) — The factory’s throughput and automation proposal is qualified by this candidate’s emphasis on maintainability, upfront alignment, and continued human ownership of code.

## Context Map

- Layer: agent
- Domains: coding
- Topics: harness-engineering, cloud-agents, agent-evals

## Uncertainty

- The talk presents Warp Factories as one implementation, but the factory approach is broader. Evaluation can use an LLM judge, which Lloyd says may be supplemented by human grading; the examples do not establish that this approach fits every team or task.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
