Open Brain
arXivPaperNeeds Review

Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients

An audit found that mental health recommendations from language models varied with whether a clinician, caregiver, or patient described the case. It is a reminder to test advice across user contexts.

arXiv · Nov 9, 2026
Open Source Open MarkdownOpen JSON
Source Summary

Researchers tested **41 language models** on **19 case vignettes**, collecting **35,055 responses** across clinician, caregiver, and self-described scenarios. They found that recommendation patterns differed by who presented the case.

Practical Implication

For systems that answer health questions, evaluate responses across user roles and check whether named interventions match the model’s own ratings. The audit found that models sometimes named options they rated below zero as initial choices when asked about a specific intervention type.

Agent-Ready Context
Researchers tested **41 language models** on **19 case vignettes**, collecting **35,055 responses** across clinician, caregiver, and self-described scenarios. They found that recommendation patterns differed by who presented the case.

For systems that answer health questions, evaluate responses across user roles and check whether named interventions match the model’s own ratings. The audit found that models sometimes named options they rated below zero as initial choices when asked about a specific intervention type.

This is an audit of recommendations, not evidence about clinical outcomes. The instrument, code, and corpus are planned for release on publication.
Connected Context · Feed7 Judgment

This Signal adds user role as a distinct axis for auditing clinical recommendations: the same case can elicit different advice for clinicians, caregivers, and patients, and a model may recommend an intervention it rates negatively. It supports checking both role-specific responses and consistency between ratings and recommendations, while leaving clinical outcomes untested.

Context Map
benchmarkresearch
Uncertainty
This is an audit of recommendations, not evidence about clinical outcomes. The instrument, code, and corpus are planned for release on publication.