# Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Source: [AI Engineer](https://www.youtube.com/watch?v=0vphxNt4wyk)  
Feed7 permalink: https://feed7.dev/p/don-t-ship-skills-without-evals-philipp-schmid-google-deepmind-0fuh3ko  
Published: 2026-07-14T22:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Agent skills need regression tests, not manual spot checks. Test triggering and output with and without each skill, across repeated trials and the harnesses your team actually uses.

## Source Summary

A cited scan indexed more than **50,000 skills** and found almost none had evals. DeepMind keeps tests beside each internal skill, runs them on every change, and checks triggers, commands, traces, and outputs with scripts or an LLM judge.

## Practical Implication

Start with positive and negative prompts, then add production traces. Run **2–6 trials per case**, test across the models and harnesses you support, and compare results with the skill enabled and removed; deterministic workflows may belong in scripts instead.

## Agent-Ready Context

A cited scan indexed more than **50,000 skills** and found almost none had evals. DeepMind keeps tests beside each internal skill, runs them on every change, and checks triggers, commands, traces, and outputs with scripts or an LLM judge.

Start with positive and negative prompts, then add production traces. Run **2–6 trials per case**, test across the models and harnesses you support, and compare results with the skill enabled and removed; deterministic workflows may belong in scripts instead.

Agent runs are nondeterministic, so a single pass proves little. Regex checks are cheap but narrow, while judge models add cost and judgment variance; retained evals remain useful after retiring a skill because they can expose later model regressions.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: skills, agent-evals, agent-reliability

## Uncertainty

- Agent runs are nondeterministic, so a single pass proves little. Regex checks are cheap but narrow, while judge models add cost and judgment variance; retained evals remain useful after retiring a skill because they can expose later model regressions.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.