PaperlayerView exact revision on arXiv

Source-linked AI summary

Pedagogical AI in Mental Health: A Tri-Stream Fine-Tuned LLM Framework for Automated Clinical Supervision and Risk Triage

Shreeya Sharma, Ravish Gupta, Saket Kumar, Abhishek Aggarwal

arXiv:2608.18438v1cs.CLcs.AIcs.LG

TL;DR

Mental-health supervision remains delayed, while research has largely focused on patient-facing AI rather than supervisory support. This paper develops a tri-stream VAL system using fine-tuned Mistral-7B for structured supervision reports and automated risk triage, achieving 95% technique identification accuracy and reducing triage latency from 72 hours to under 15 seconds.

  • Problem

    Mental-health AI research has largely neglected the supervisory layer, where delayed review and limited oversight constrain support for trainees managing clinical risk.

  • Method

    A fine-tuned Mistral-7B analyzes therapy sessions across visual, acoustic, and linguistic streams to generate supervision reports and route risk through the D-CUI.

  • Results

    95% technique identification accuracy and triage latency below 15 seconds were achieved on 106 DAIC-WOZ sessions.

  • Takeaways & Limitations

    The system supports automated screening that flags sessions needing immediate attention and provides structured evidence for supervisors’ clinical judgment.

  • Takeaways & Limitations

    Evaluation is limited to 22 test sessions, two techniques, and semi-structured virtual-agent interviews, so performance cannot yet generalize to naturalistic therapy.

Abstract

from arXiv · show

Modern mental healthcare faces a critical shortage of senior supervisory oversight, leading to a "supervision gap" where novice therapists manage high-stakes risks with delayed professional feedback. This paper proposes a new framework utilizing a fine-tuned Mistral-7B-instruct model as an automated "Supervisor-in-the-Loop" system. By leveraging 106 sessions from the DAIC-WOZ dataset, the model performs a tri-stream analysis: (1) Therapeutic Alliance tracking via semantic adherence, (2) Latent risk prediction using attention-weighted analytics, and (3) Supervisory Triage via a Dynamic Clinical Urgency Index (D-CUI). Our multi-modal VAL (Visual-Acoustic-Linguistic) framework achieves 95% technique identification accuracy [95% CI: 75.1%-99.9%], alliance assessment MAE of 0.105 on a 5-point scale [95% CI: 0.059-0.151], therapeutic fidelity alpha = 0.423, and mean D-CUI of 0.370 [95% CI: 0.322-0.419]. Training converged in 105 steps with 85.2% loss reduction on a single Tesla T4 GPU. The system reduces supervisory triage latency from 72 hours to real time (~10 seconds per session), enabling proactive intervention in high-risk cases. The system addresses the cold-start problem through Bayesian priors and implements timestamp-based modality synchronization for robust multi-modal fusion.

1 Introduction

Clinical supervision is often delayed by weekly review structures, leaving urgent risks unresolved while supervisor-to-trainee ratios exceed 1:12. This paper addresses the largely neglected supervision layer with an automated supervisor-in-the-loop that analyzes therapy sessions across VAL channels and produces risk-prioritized reports.

  • Clinical supervision gap: The supervision gap can leave suicidality disclosures, trauma revelations, and therapeutic ruptures unreviewed for 72 hours, while supervisor-to-trainee ratios often exceed 1:12.Traditional retrospective case review supports professional development but was not designed for urgent clinical needs.
  • Research gap: Research has focused LLMs on patient-facing tools, while largely neglecting supervision, where quality failures, trainee risk handling, and structural capacity pressures converge.Examples include psychoeducation chatbots, documentation drafters, and CBT-scripted conversational agents.
  • Proposed system: The proposed system treats the LLM as an automated supervisor-in-the-loop that processes therapy sessions through Visual, Acoustic, and Linguistic channels and generates structured, risk-prioritized supervision reports.The work builds on earlier group research addressing explainability for clinical AI decisions and identity and access management for healthcare AI systems.
  • Tri-stream architecture: The tri-stream VAL architecture simultaneously processes 100Hz COVAREP audio, 30fps OpenFace video, and transcripts, aligning modalities by snapping them to utterance boundaries.The design addresses synchronization across the three sampling rates.
  • Risk-prioritized triage: The D-CUI routes supervision using risk probability, symptom severity, inverted therapist experience, and session-level sentiment volatility as inputs.Inverting therapist experience makes inexperience amplify urgency.

2 Related Work

Mental-health AI has progressed from scripted CBT chatbots to multimodal detection and LLM-based therapeutic responses, but prior systems remain focused on patient-side tasks rather than clinical supervision. DAIC-WOZ work likewise emphasizes depression classification, motivating a temporally aligned VAL framework for supervisory analysis.

  • Related Work: Prior systems span scripted CBT, speech-based depression detection, transformer suicide-risk screening, and LLM-generated therapy responses [10].Woebot launched in 2017 as a scripted CBT chatbot, while later systems expanded toward deep learning, transformers, and therapist-like LLM responses.
  • Supervision Gap: The literature largely targets patient-side screening, conversation, monitoring, or documentation, leaving the supervisory layer governing clinical quality unaddressed.Existing evaluations emphasize user satisfaction and symptom reduction rather than supervisory review and clinical governance.
  • Dataset and Prior Benchmarking: DAIC-WOZ provides semi-structured multimodal interviews from 189 participants with audio, video, transcripts, and PHQ-8 annotations.The corpus is a multimodal benchmark for mental-health AI, but prior work has concentrated mainly on classifying depression and its severity.
  • Multimodal Fusion: Text-only depression detection on DAIC-WOZ reaches F1 = 0.67, audio raises it to 0.77, and visual input adds further incremental benefit.These gains are constrained by alignment challenges among 100Hz audio, 30fps video, and utterance-level text, where fusion misalignment can introduce substantial noise.
  • Methodological Motivation: The proposed VAL framework uses timestamp-based windowing to preserve clinically meaningful temporal context while avoiding the GPU-memory demands of frame-level cross-modal attention.This deliberate trade-off distinguishes events by session timing, such as acoustic distress at minute 23 versus minute 3.

3 System Architecture · 3.1 Overview · 3.2 The VAL Ingestion Engine

The system uses a tri-stream VAL architecture to extract visual, acoustic, and linguistic signals, fuse session representations, and route cases through the D-CUI. Its staged pipeline aligns inputs before fidelity analysis, risk triage, and supervisory guidance generation.

  • 3.1 Overview: The Visual stream analyzes OpenFace facial Action Units at 30 fps to detect affect, engagement, and incongruent distress signals.Safety-critical incongruence includes verbal distress denial alongside facial distress markers such as AU1 and AU4 without AU12.
  • 3.1 Overview: The Acoustic stream processes COVAREP features at 100 Hz, including F0 variability and Voice Unvoiced mean, to capture expressivity, fluency, pauses, flat affect, and agitation.These paralinguistic signals capture states that transcript text alone may miss.
  • 3.1 Overview: The Linguistic stream applies fine-tuned Mistral-7B-instruct to transcripts to identify therapeutic techniques, alliance markers, suicidal ideation, and hopelessness language.Examples of detected techniques include Reflective Listening and Open-Ended Questioning.
  • 3.1 Overview: VAL processes therapy sessions through Visual, Acoustic, and Linguistic streams, fuses outputs at utterance boundaries, and routes cases via the D-CUI to escalation or routine review.The architecture analyzes fidelity, affect incongruence, and clinical risk before routing supervisory outcomes.
  • 3.1 Overview: The D-CUI combines patient risk, symptom severity, therapist experience, and within-session sentiment volatility into one urgency score for escalation or routine review.Fused session-level representations from all three streams feed this routing calculation.
  • 3.2 The VAL Ingestion Engine: The four-stage pipeline aligns and normalizes streams, scores therapeutic fidelity, computes the D-CUI, and generates supervisory guidance with routing.Sequencing prevents noise from misaligned fidelity analysis and ensures triage precedes report generation.

Modality Synchronization and Temporal Alignment … Visual Stream

The framework synchronizes acoustic and visual streams to transcript-defined utterance windows, then uses Mistral-7B to analyze therapeutic techniques, alliance, and risk. Visual incongruence detection specifically amplifies supervisory urgency when verbal reassurance conflicts with facial distress.

  • Modality Synchronization and Temporal Alignment: Transcript-defined utterance boundaries align 100Hz COVAREP audio and 30fps OpenFace video, preserving clinically relevant temporal structure through lossy aggregation.Both streams are snapped to DAIC-WOZ utterance start/stop markers rather than averaged across entire sessions.
  • Modality Synchronization and Temporal Alignment: Acoustic means, Action Unit means, and raw transcript text are concatenated into session vectors and serialized as natural language for Mistral-7B.For each turn t_i, acoustic and visual features are mean-pooled over [s_i:e_i].
  • Linguistic Stream: Mistral-7B tags Reflective Listening, Open-Ended Questioning, alliance markers, and risks including suicidal ideation, self-harm, hopelessness, and safety concerns.Alliance markers include collaborative language, empathy, goal consensus, and rupture indicators.
  • Acoustic Stream: COVAREP contributes F0 standard deviation and Voice Unvoiced mean to capture pitch variability, speech continuity, silence, hesitation, and fragmentation.Low F0 variance correlates with flat affect in depression, while elevated Voice Unvoiced values indicate hesitation or fragmentation.
  • Visual Stream: OpenFace tracks distress, engagement, and incongruent-affect indicators using clinically relevant Action Units, gaze direction, and verbal–facial mismatches.Distress includes AU1, AU4, and AU15; engagement includes AU12, AU26, and gaze direction.
  • Visual Stream: Incongruence fires when VADER sentiment exceeds 0.3, AU01 or AU04 exceeds 0.5, and AU12 remains below 0.3.This rule targets verbal denial of distress alongside facial brow-raising or brow-lowering without a smile signal.
  • Visual Stream: When incongruent affect fires, Δϕ=0.4 multiplicatively amplifies D-CUI through (1+Δϕ), triggering supervisory review despite moderate other risk indicators.The visual stream’s safety-critical role is identifying patients whose facial distress conflicts with verbal reassurance.

3.3 Therapeutic Fidelity Analyzer · 3.4 Risk Triage Module · Dynamic Clinical Urgency Index (D-CUI)

The Therapeutic Fidelity Analyzer quantifies adherence to evidence-based methods, while the D-CUI prioritizes sessions for supervisory review using risk, symptom severity, therapist experience, and sentiment volatility. Bayesian initialization and experience updating extend triage to therapists with limited historical data.

  • 3.3 Therapeutic Fidelity Analyzer: Therapeutic Fidelity (α) is defined as cosine similarity between session embeddings (VT) and clinical manual embeddings (VM).
  • 3.3 Therapeutic Fidelity Analyzer: Fidelity scores above 0.5 indicate close adherence, scores from 0.3 to 0.5 indicate partial adherence, and scores below 0.3 flag deviation for supervisory attention.
  • Dynamic Clinical Urgency Index (D-CUI): The D-CUI determines which sessions require immediate supervisory attention versus routine review by combining four clinical and supervisory factors.
  • Dynamic Clinical Urgency Index (D-CUI): D-CUI = (w1R + w2S + w3(1 − E)) · (1 + Δϕ), with empirically tuned weights w1 = 0.40, w2 = 0.35, and w3 = 0.25.
  • Dynamic Clinical Urgency Index (D-CUI): Risk probability, PHQ-8-derived symptom severity, therapist experience, and within-session sentiment volatility define the D-CUI inputs.R, S, and E range from 0 to 1; Δϕ captures emotional instability, and E represents normalized years of practice.
  • 3.4 Risk Triage Module: The (1 − E) term increases urgency for high-risk clients treated by novice therapists, distinguishing supervisory situations that patient-only risk models overlook.
  • Dynamic Clinical Urgency Index (D-CUI): When therapist experience is unavailable, the system initializes E0 = 0.3 as an early-career prior and updates E through an exponential moving average as session data accumulates.The prior intentionally favors broader supervisory coverage.

Addressing the Cold-Start Problem · Sentiment Volatility (∆ϕ) · 3.5 Supervisory Guidance Generator

The framework addresses cold-start supervision with β = 0.7 smoothing once therapist experience becomes available, quantifies sentiment volatility through sequential emotional change, and maps D-CUI scores to tiered supervisory guidance. Immediate cases trigger transcript-grounded alerts and safety protocols, while elevated cases receive prioritized review.

  • Addressing the Cold-Start Problem: Cold-start smoothing uses β = 0.7, with Eobserved representing the therapist’s documented experience level once available.The smoothing factor supports experience-aware initialization before observed experience is available.
  • Addressing the Cold-Start Problem: The cold-start mechanism updates supervision using documented therapist experience when Eobserved becomes available.This condition distinguishes the initialization phase from later experience-informed estimation.
  • Sentiment Volatility (∆ϕ): Sentiment volatility captures rapid emotional shifts within a session that may indicate crisis states or therapeutic ruptures.It is computed as the mean absolute rate of change in sentence-level sentiment scores.
  • Sentiment Volatility (∆ϕ): The ∆ϕ calculation emphasizes sequential emotional transitions rather than overall sentiment dispersion.ϕ_i denotes the sentiment score for sentence i.
  • 3.5 Supervisory Guidance Generator: D-CUI exceeds 0.7 to trigger an Immediate alert containing transcript lines that drove escalation and a mapped safety protocol.The alert provides actual source material rather than a summary, allowing supervisory disagreement before action.
  • 3.5 Supervisory Guidance Generator: D-CUI between 0.4 and 0.7 is classified as Elevated and flagged for prioritized review within the current supervisory cycle.This tier is above routine review but is not treated as an emergency.

4 Implementation

The implementation uses Mistral-7B-instruct [18] with QLoRA [19] to support deployable, auditable supervision on accessible clinical hardware. It prepares synchronized multimodal training data and applies privacy, uncertainty, and high-risk routing safeguards.

  • Model Selection: Mistral-7B-instruct [18] fits institutional deployment constraints through 4-bit quantization, Apache 2.0 on-premise operation, and instruction-following outputs suited to templated supervision reports.It requires approximately 2 GB of storage and 5.6 GB of inference GPU memory, fitting a Tesla T4, while supporting auditability and keeping transcripts within institutional infrastructure.
  • Data Preparation: The pipeline diarizes recordings, extracts acoustic and visual features, aligns streams to utterance boundaries, and converts rule-labeled synthetic supervision examples into Alpaca instruction-tuning format.Labels cover therapeutic techniques, alliance markers, and clinical concerns; additional prompts and responses were generated synthetically using Claude Sonnet 4.6 Thinking.
  • Training Strategy: QLoRA [19] fine-tuning uses a 2e-4 learning rate, batch size 4, rank 64, alpha 16, five epochs, and completes in 105 steps on one Tesla T4.Training took 2 hours 44 minutes with the paged AdamW 32-bit optimizer.
  • Privacy and Governance: The system strips identifying information, encrypts storage and transmission end-to-end, role-gates access, logs every access event, and discloses AI involvement before sessions.These controls address the sensitivity of therapy disclosures and restrict session-data access to supervising clinicians.
  • Safety Constraints: Low-confidence outputs trigger uncertainty deferral, while suicidal ideation or abuse disclosures bypass triage and generate immediate alerts.The routing design avoids authoritative-looking recommendations when confidence is low and excludes urgent safety signals from automated queue handling.

5 Experimental Evaluation

On 22 held-out DAIC-WOZ sessions, the model showed strong therapeutic-technique recognition and low alliance-assessment error, while routing supervisory triage from a 72-hour queue to near real time. D-CUI incorporated affect incongruence to elevate clinically important cases, although this adjustment was minimal in routine screening sessions.

  • 5 Experimental Evaluation: The evaluation used 106 DAIC-WOZ sessions, with 84 training sessions and 22 held-out test sessions selected using random seed 42.The evaluation tested held-out therapeutic-technique identification and faster routing of urgent cases than traditional weekly supervision review.
  • 5 Experimental Evaluation: 85.2% cross-entropy loss reduction accompanied convergence in 105 steps, and end-to-end processing completed in under 15 seconds per session instead of a 72-hour supervisory queue.Training took 2 hours 44 minutes on a Tesla T4; analysis required 8–12 seconds and D-CUI computation under 1 millisecond.
  • 5 Experimental Evaluation: D-CUI increased from 0.283 to 0.395 for a low-risk session when affect incongruence was set to ∆ϕ = 0.4, moving it near the Elevated threshold.Without the visual stream, the session remained within the Routine band; the D-CUI raw score is clamped to [0, 1].
  • 5 Experimental Evaluation: In the DAIC-WOZ corpus, ∆ϕ ranged from 0.00 to 0.02, producing less than 2% D-CUI adjustment because screening dynamics were stable.The ∆ϕ term is designed to activate in acute scenarios rather than routine screening.

6 Discussion

The framework is positioned as an automated screening aid that flags sessions for supervisory attention, not as a replacement for clinical supervision. Preliminary expert ratings support report intelligibility, while limited data, narrow scope, privacy risks, and deployment constraints require cautious interpretation.

  • Practical implication: The system is intended to flag the 10–15% of sessions needing immediate attention and provide structured evidence, freeing supervisors for clinical judgment rather than routine monitoring.The 95% technique-identification result is a proof of concept from two techniques and 22 sessions, not evidence of clinical readiness.
  • Novelty: The D-CUI incorporates therapist experience alongside patient risk factors, treating supervisory urgency as dependent on both client risk and clinician experience.This distinguishes situations involving the same high-risk client but supervisors with substantially different experience levels.
  • Multimodal contribution: Multimodal analysis detects incongruent affect through facial markers and clinically meaningful acoustic cues such as flat affect and speech fragmentation that transcripts cannot capture.The visual stream can contradict verbal content, while the D-CUI amplifies urgency; low F0 variance and elevated VUV contribute distinct acoustic evidence.
  • Limitations and future work: The evidence is bounded by 22 semi-structured DAIC-WOZ sessions, two techniques, one Mistral-7B model, isolated sessions without longitudinal context, and CBT-oriented training data.The authors call for naturalistic therapy data, broader technique taxonomies, longitudinal tracking, and real-time monitoring integration.
  • Clinical usability: 4.2/5.0 clinical relevance, 3.9/5.0 triage appropriateness, and 4.0/5.0 actionability were reported by five supervisors reviewing 22 reports, with Cohen’s κ = 0.61.The evaluation provides preliminary evidence of clinical intelligibility but was limited by its small sample and controlled setting.
  • Ethics and deployment: Deployment requires disclosure, HIPAA verification, anonymization, encryption, role-gated access, meaningful supervisor review, bias monitoring, and a genuine client opt-out path.AI processing may create new exposure points, perform unevenly across populations, and alter how clients speak during therapy.

7 Conclusion

The tri-stream fine-tuned Mistral-7B system analyzes therapy sessions across visual, acoustic, and linguistic modalities, producing structured supervision reports and automated risk triage. Across 106 DAIC-WOZ sessions, it achieves 95% technique identification accuracy and reduces triage latency from 72 hours to under 15 seconds, while remaining a proof-of-concept.

  • Conclusion: 95% technique identification accuracy and under 15-second triage latency demonstrate the system’s core contribution across 106 DAIC-WOZ sessions.The system analyzes visual, acoustic, and linguistic modalities to produce structured supervision reports with automated risk triage.
  • Conclusion: The evaluation remains a proof-of-concept based on 22 test sessions, two techniques, and one dataset of semi-structured interviews.The passage states that these constraints require further attention.
  • Conclusion: By surfacing sessions needing immediate attention in seconds rather than days, the system addresses a structural supervision problem while leaving clinical judgment, empathy, and accountability with supervisors.It removes monitoring burden rather than replacing supervisory responsibility.

Disclosure of Interests

The authors state that the work was conducted independently, does not represent or relate to their listed organizations, and has no relevant competing interests.

  • The work was conducted independently.
  • It does not relate to the authors’ positions at Microsoft, BigCommerce, or Amazon, nor represent those organizations’ views or interests.
  • The authors declare no competing interests relevant to the article.
Loading 2608.18438v1…