Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions

• Almost half of people with mental health conditions have used AI chatbots for psychological support.1–3Source: Introduction · p. 1
• Reported high-stakes failures and an emerging patchwork of state restrictions underscore the lack of an operational framework for safe deployment.4;5;6Source: Introduction · p. 1
• The authors describe human-on-the-loop oversight informed by a tool used across 352,649 conversations, combining automated and human evaluation to maintain quality and safety at scale.Source: Introduction · p. 1

• Even motivated reviewers struggle to sustain attention to rare events for more than 30 minutes, making review of hundreds of AI conversations vulnerable to missed failures.10Source: Beyond "Human-in-the-Loop": Oversight… · p. 2
• Human-on-the-loop oversight focuses attention on likely-to-need-help cases and shifts clinicians toward quality metrics, flagged concerns, and improving the evaluation system.The paper describes this model as mitigating fatigue and automation bias.Source: Beyond "Human-in-the-Loop": Oversight… · p. 2
• The authors introduce a three-layer oversight process based on deploying and monitoring an AI coaching tool at scale.Source: Beyond "Human-in-the-Loop": Oversight… · p. 2

Layer 1: Preventive Design

• Layer 1 defines the coach’s scope as reflection and between-session practice, while directing therapy, diagnosis, prescribing, and crisis intervention to clinicians or appropriate resources.Its clinical foundation draws on evidence-based therapies and prompts developed by licensed clinical psychologists.Source: Layer 1: Preventive Design · pp. 2–3
• Pre-deployment testing combines multi-turn clinical benchmarks, therapist feedback, pilot testing, LLM-as-judge validation, A/B tests, and red-teaming.In the mock benchmark examples, one conversation scores Good across eight dimensions, while another is Unacceptable on three.Source: Layer 1: Preventive Design · p. 3, Figure 1 · p. 4

Layer 2: Real-Time Safety Monitoring

• Layer 2 processes messages with parallel coaching and safety evaluators, then uses contextual risk assessment to select check-ins or escalation.Higher concern can pause the conversation, display crisis resources, and notify the treating provider.Source: Figure 2 · p. 5, Layer 2: Real-Time Safety Monitoring · p. 5
• Providers can view conversations, receive safety alerts, and enable or disable coaching, but individual-event monitoring cannot establish system-wide performance over time.The paper identifies continuous quality assurance as the layer needed to address that broader question.Source: Layer 2: Real-Time Safety Monitoring · pp. 5–6

Layer 3: Continuous Quality Assurance

• 61.9% of conversations showed a shift toward more positive or hopeful language, and 64.9% showed greater motivation, commitment, or readiness to act by the end.Automated scoring covers every conversation across dimensions including empathy, safety, clinical appropriateness, and effectiveness.Source: Layer 3: Continuous Quality Assurance · p. 6
• Clinicians review a triaged subset, prioritizing safety events and lower-quality or unusually long conversations while sampling unflagged conversations for calibration.Review findings inform threshold, prompt, and evaluator updates across the framework.Source: Layer 3: Continuous Quality Assurance · p. 6, Table 1 · p. 7
• Reviewers found panic-like symptoms were overcategorized, and recalibrating safety thresholds reduced unnecessary interruptions without compromising detection of genuine clinical concern.Subsequent evaluation confirmed the reduction in overtriggering.Source: Layer 3: Continuous Quality Assurance · p. 6
• Reviewers traced sycophantic responses in interpersonal situations to accumulated client framing in memory, and a shorter context window improved the non-sycophantic dimension.Source: Layer 3: Continuous Quality Assurance · pp. 6–7

• The three-layer validation framework was deployed across approximately 350,000 conversations over ten months.Source: What the Data Shows: Lessons from a… · p. 7
• Approximately 4× more often, conversations flagged as lower quality by the automated evaluator were confirmed as lower quality by human reviewers than unflagged conversations.The automated layer scored every conversation and created a priority-ranked pool for human review.Source: What the Data Shows: Lessons from a… · p. 7
• When real-time safety had intervened on a flagged conversation, clinicians were less likely to rate it as problematic, often assessing it after the conversation had shifted toward the client’s needs.Reviewers then assessed whether the intervention was appropriate.Source: What the Data Shows: Lessons from a… · p. 7
• Clinician findings informed system-prompt updates and recalibration of real-time safety thresholds, making the oversight layers more effective over time.Source: What the Data Shows: Lessons from a… · p. 7

Implications for Mental Health Professionals

• The framework was deployed on one product at one company, and the authors offer its principles for professionals evaluating AI systems used in clinical care.Source: Implications for Mental Health… · p. 8

Ask about clinical input at multiple levels

• Ask what clinical expertise shaped system design, how risk is detected and handled in real time, and how clinicians evaluate performance over time.Relying on only one oversight layer may make a system more susceptible to mistakes.Source: Ask about clinical input at multiple… · p. 8

Understand the role of clinician review

• Professionals should ask how often conversations receive human review, how they are selected, and what happens when reviewers find problems.Reviewing every message is impractical for sustained attention and can lead to review-fatigue errors.Source: Understand the role of clinician review · p. 8

Look for the feedback loop

• Look for a feedback loop in which reviewer findings refine decision thresholds and AI responses, with increasing agreement between automated and human evaluation.Without iterative refinement, oversight monitors the system without helping it improve.Source: Look for the feedback loop · p. 8

Advocate for transparency

• Professionals should advocate for transparency about oversight processes and contribute clinical input as regulatory frameworks develop.The boundaries of appropriate AI use in mental healthcare are still being defined.Source: Advocate for transparency · p. 8

• The paper frames the practical challenge as evaluating AI systems so they are as safe, clinically appropriate, and accountable as possible while their use grows.Source: Conclusion · p. 8
• The framework combines preventive design, real-time monitoring, and continuous evaluation through a feedback loop connecting all three layers.Source: Conclusion · pp. 8–9
• Across hundreds of thousands of conversations, automated evaluation provided comprehensive coverage and triage, while clinicians added nuanced judgment and responsibility that automation cannot fully replicate.The feedback loop connecting the two allows the system to improve over time.Source: Conclusion · p. 9
• The authors argue that the most useful oversight is not necessarily the most extensive, but concentrates human expertise where it matters and lets oversight layers reinforce one another.They also call for a framework that improves with each reviewed conversation.Source: Conclusion · p. 9

• M.A.S., K.R., and M.K. are Grow Therapy employees; E.G. advises the company, and J.L.H. collaborates on ongoing research with it.Source: Competing Interests · p. 9
• Grow Therapy develops and operates the AI coaching tool described in the manuscript.Source: Competing Interests · p. 9
• The disclosed relationships include ongoing research collaboration with Grow Therapy.Source: Competing Interests · p. 9