Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop
Agent evals tied to a model, tool call, or harness can decay after an upgrade. Keep checks in code, prioritize production failures by onset and reach, and use agents to investigate detected anomalies.
Hylak says a harness switch made **80% of one eval suite** unhelpful. He recommends running **local, code-based agent tests** instead of relying on prompt playgrounds or large suites coupled to current tool behavior.
Treat production evidence as the durable layer. For each failure class, track **when it began** and **the share of users affected**, then use those signals to decide what deserves a regression test or rollback investigation.
Hylak says a harness switch made **80% of one eval suite** unhelpful. He recommends running **local, code-based agent tests** instead of relying on prompt playgrounds or large suites coupled to current tool behavior. Treat production evidence as the durable layer. For each failure class, track **when it began** and **the share of users affected**, then use those signals to decide what deserves a regression test or rollback investigation. Agents are weak anomaly detectors, and free-form clustering can blur distinct root causes. Detect measurable shifts such as keyword-frequency spikes first, then ask an agent to investigate them.
This makes production failures and measurable shifts the durable basis of agent evaluation, while treating harness-coupled suites as disposable. It sharpens the prior trace-and-simulation loop: detect anomalies with explicit metrics, preserve onset and affected-user scope, then promote verified failure classes into small local regression tests rather than asking agents to discover structure through free-form clustering.