AI EngineerVideoSource Linked
Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind
SonderMind turns clinician-reviewed failures into release-gating evals, keeping mental-health guardrails modular and testing false positives, false negatives, category, and timing.
AI Engineer · Jul 26, 2026
Source Summary
SonderMind built **separate input and output guardrail LLMs** around its coaching model instead of relying on general-purpose moderation. Flagged conversations are reviewed by clinicians, normalized into **typed evals**, and committed as **CI release gates**.
Practical Implication
For high-stakes agents, trace ambiguous failures and make domain experts responsible for defining acceptable behavior. Evaluate whether the correct category fired at the correct conversational moment, including both false negatives and harmful over-triggering.
Agent-Ready Context
SonderMind built **separate input and output guardrail LLMs** around its coaching model instead of relying on general-purpose moderation. Flagged conversations are reviewed by clinicians, normalized into **typed evals**, and committed as **CI release gates**. For high-stakes agents, trace ambiguous failures and make domain experts responsible for defining acceptable behavior. Evaluate whether the correct category fired at the correct conversational moment, including both false negatives and harmful over-triggering. The talk provides an engineering process, not measured safety rates or a claim of perfect detection. Clinical language can remain ambiguous, separate judge calls add latency and cost, and benchmark optimization can drift away from the people the tests are meant to protect.
Context Map
benchmark#agent-evals#agent-reliabilityUncertainty
The talk provides an engineering process, not measured safety rates or a claim of perfect detection. Clinical language can remain ambiguous, separate judge calls add latency and cost, and benchmark optimization can drift away from the people the tests are meant to protect.