Source-linked AI summary
Augmenting Interviewer Judgments of Patient Experience with Automatic Language Analysis
Aowen Shi, Michal Balazia, Danilo Postin, René Hurlemann, Jan Alexandersson, François Brémond, Philipp Müller
TL;DR
Interviewers’ judgments of psychiatric patients’ clinical experiences do not always match patients’ self-reports, and it was unclear whether automatic language analysis could complement human judgment. The paper combines interviewer ratings with language-based predictions across five models and finds that simple averaging provides the strongest overall performance, indicating complementary information sources.
Problem
Interviewers may inaccurately judge patients’ experienced interaction quality, while it remained unclear whether automatic conversation-quality predictions complement rather than replicate human judgments.
Method
The study combines post-session interviewer ratings with predictions from five language-based models using separately extracted patient- and interviewer-stream sentence embeddings from 107 free clinical interviews.
Results
Interviewer integration achieved Pearson r = 0.403 ± 0.030, outperforming the interviewer-only baseline (r = 0.365) and the best fully automatic model (r = 0.286 ± 0.044).
Takeaways & Limitations
Automatic language analysis is most useful as an additional evidence source that enhances rather than replaces interviewer judgment when approximating patients’ perspectives.
Takeaways & Limitations
Under the present modeling setup, dual-stream inputs did not consistently improve over interviewer-only inputs.
Abstract
from arXiv · showhide
Understanding how psychiatric patients subjectively experienced a clinical conversation is important for feedback and alliance-related process monitoring. While interviewers form post-session judgments about patient experience, these judgments do not always match patients' self-reports. Automatic approaches for predicting perceived interaction quality from conversation have been proposed, but it remains unclear whether such approaches can complement human judgment rather than simply replicate it. To address this gap, we evaluate a clinician-support framework in which post-session interviewer ratings are combined with automatic language-based predictions to estimate patient-reported interaction quality in free clinical interviews. We assess this integration across multiple standard model types, including Ridge, SVR, MLP, GRU, and BiLSTM, all trained on sentence embeddings extracted from dyadic transcripts of 107 free conversations between psychiatric patients and interviewers. Our results show that combining interviewer judgments with model predictions through simple averaging yields the strongest overall performance. The interviewer-only baseline reached a Pearson correlation of 0.365. Among fully automatic models, Ridge achieved the strongest Pearson correlation (r = 0.286), while BiLSTM achieved r = 0.270. The strongest result was obtained by BiLSTM interviewer integration (r = 0.403). Our findings suggest that automatic language analysis and interviewer judgment capture complementary aspects of patient experience and that their combination provides a more accurate approximation of the patient's own report than either source alone.
I. INTRODUCTION
Patient experience is clinically important but can diverge from interviewer judgments. This study examines whether automatic language analysis can complement those judgments when estimating patient-reported interaction quality.
- Patient experience is linked to satisfaction, therapy outcomes, and therapeutic alliance, making its assessment important in clinical care.
- Clinical personnel may inaccurately judge patients’ experienced interaction quality, including underestimating perceived alliance and overestimating positive emotions.
- Automatic conversation-quality approaches have been studied separately from human judgments, leaving their complementarity unclear.
- The study combines language-based predictions from five model types with interviewer judgments by averaging the resulting predictions.
- r = 0.403 ± 0.030 for interviewer integration, compared with r = 0.365 for interviewer-only judgments and r = 0.286 ± 0.044 for the best fully automatic model.
B. Prediction of interaction quality from conversation
Prior work shows that interaction quality can be modeled from behavioral and spoken-language signals, while multimodal methods introduce practical constraints. This study focuses on scalable, privacy-preserving language-based modeling in psychiatric clinical sessions.
- Automatic interaction-quality research has modeled rapport, engagement, and perceived conversation quality across natural-interaction settings and behavior modalities.
- Psychotherapy studies indicate that lexical content and spoken interaction contain measurable signals of session quality and therapist skills.
- The open question is whether automatic estimates contain information independent of a human observer’s judgment of the same interaction.
- Language-based modeling is emphasized as a more scalable and privacy-preserving starting point than approaches requiring audio and video.
- The analysis uses 107 free-format dyadic clinical sessions involving psychiatric patients meeting criteria for a Major Depressive Episode.
B. Interaction quality measure
The study measures free-interview experience with three parallel patient-report and interviewer-estimate questionnaire items, while treating their composite as related to but not equivalent to therapeutic alliance.
- The analyzed questionnaire subset contains three items available in both patient-report and interviewer-estimate form: helpfulness, ease of sharing, and mood change.
- For each session, the primary target is the mean of the three patient-side items measuring overall patient-reported experience.
- The patient-side items assess feeling better, conversation helpfulness, and ease of sharing personal information.
- The interviewer composite averages corresponding interviewer-side items and captures mood improvement, helpfulness, and relational openness.
- 106 sessions were analyzed because one of 107 sessions lacked the interviewer questionnaire; patient and interviewer means were 69.5 and 66.1, respectively.
- Both composite distributions were left-skewed, with patients reporting slightly higher average scores than interviewers.
IV. METHODS
The framework extracts separate patient and interviewer language representations, predicts patient experience with pooled or sequence regression models, and averages model predictions with interviewer ratings.
- Each session is represented separately for patient and interviewer streams before regression models predict patient-reported experience.
- Transcripts are produced from separate speaker channels and segmented into overlapping 30-second windows with a 10-second stride.
- Pooled models compress each speaker stream into a session-level vector, whereas sequence models preserve temporal structure before pooling.
- Pooled models: For pooled Ridge and SVR models, each speaker stream is mean-pooled directly from window embeddings.
- Pooled models: The MLP applies a shared window encoder before mean-pooling encoded features within each speaker stream.
- Sequence models: Sequence models separately encode ordered patient and interviewer streams, apply masked mean pooling, fuse them late, and predict a scalar.
C. Prediction settings
The study compares raw interviewer ratings, fully automatic predictions, and interviewer-integrated predictions for estimating patient-reported interaction quality. Integration combines automatic model output with the interviewer’s rating through fixed averaging.
- Raw interviewer ratings are used directly as a proxy for the patient’s report and as the human-judgment baseline.
- Fully automatic predictions are produced by family-specific regressors applied to session-level representations.Ridge and SVR use pooled representations, MLP uses its corresponding representation, and GRU and BiLSTM use sequence representations.
- Interviewer integration combines the automatic prediction with the raw interviewer rating using a fixed arithmetic operation.
- The fusion requires no additional learned parameters and is computed on held-out sessions after automatic-model training within the corresponding training folds.
A. Training setup and evaluation metrics
The evaluation uses participant-level nested cross-validation, fixed inner-fold hyperparameter searches, repeated randomized runs, and Pearson correlation with MAE as complementary metrics.
- 106 sessions with complete patient and interviewer ratings are used so all model results are directly comparable.
- Training setup: Participant-level nested GroupKFold uses 5 outer folds and 4 inner folds, keeping outer test participants unseen during evaluation.
- Training setup: Hyperparameter selection is confined to inner training folds, with search spaces fixed before the reported nested-cross-validation experiments.Neural models use AdamW, early stopping, and up to 80 epochs.
- Variance estimation: Nested cross-validation is repeated over 40 randomized runs, with varied split seeds and additional neural-network initialization seeds.
- Evaluation metrics: Pearson r is the primary metric, while MAE summarizes absolute predictive error; means and standard deviations are reported across runs.Run-level bootstrap confidence intervals and paired Wilcoxon signed-rank tests are additionally reported for main Pearson-correlation comparisons.
B. Overall Results
Interviewer integration provides the strongest overall prediction of patient-reported clinical interview experience, outperforming both interviewer-only and fully automatic approaches. The results also show positive but imperfect associations and range compression in the predictions.
- The raw interviewer rating reaches r = 0.365 and MAE = 15.717, providing a competitive human-judgment baseline.
- Ridge achieves the best fully automatic Pearson correlation at r = 0.286 ± 0.044, while SVR achieves the lowest fully automatic MAE at 15.125 ± 0.466.
- BiLSTM interviewer integration achieves the highest Pearson correlation at r = 0.403 ± 0.030 and the lowest MAE at 14.214 ± 0.312.Its run-level bootstrap 95% CI for Pearson r is [0.393, 0.412].
- Integration improves over interviewer-only predictions by ∆r = +0.0378 and over fully automatic BiLSTM by ∆r = +0.1330.The paired Wilcoxon tests report p = 3.92×10−9 and p = 9.09×10−13, respectively.
- The scatter plots show positive associations for interviewer-only and BiLSTM predictions, with stronger range compression in the fully automatic model.The integrated BiLSTM panel shows improved agreement with patient-reported ratings.
C. Speaker Ablation
Speaker-stream ablations compare patient-only, interviewer-only, and dual-stream inputs for Ridge and BiLSTM. Interviewer-only inputs produce the strongest Pearson correlations, whereas patient-only inputs are weakest.
- Speaker Ablation: The ablation evaluates patient-only, interviewer-only, and dual-stream inputs for Ridge and BiLSTM in the fully automatic setting.Dual-stream inputs combine both speaker streams by late fusion.
- Speaker Ablation: Ridge interviewer-only input reaches r = 0.298 ± 0.040, compared with r = 0.286 ± 0.044 for dual-stream and r = 0.192 ± 0.069 for patient-only input.
- Speaker Ablation: Patient-only models are weakest across both model types, while interviewer-only inputs achieve the strongest Pearson correlations.MAE shows a similar Ridge pattern, although BiLSTM dual-stream input has slightly lower MAE than interviewer-only input.
- Speaker Ablation: BiLSTM interviewer-only input reaches r = 0.285 ± 0.106, compared with r = 0.270 ± 0.084 for dual-stream and r = 0.079 ± 0.119 for patient-only input.
D. Pooling Strategy Ablation
Masked mean pooling was selected for BiLSTM results because it matched or slightly outperformed attention-based alternatives. The broader findings indicate that integrating interviewer ratings with language predictions improves estimation of patient-reported experience, with interviewer-side language providing the strongest automatic signal.
- Pooling Strategy Ablation: Masked mean pooling consistently matched or outperformed attention pooling and joint attention pooling in both BiLSTM settings.It was therefore used in the remaining BiLSTM results.
- Main Findings: Interviewer integration consistently outperformed the raw interviewer baseline and corresponding fully automatic variants across model types.The results support complementary information from interviewer judgments and language-based predictions.
- Speaker-Stream Analysis: Interviewer-side language was more informative than patient-side language for fully automatic prediction, while dual-stream inputs did not consistently improve over interviewer-only inputs.The two speaker streams were therefore asymmetric in informativeness under the reported modeling setup.
- Clinical Support: A fixed arithmetic average provides a simple way to use automatic prediction as an additional evidence source rather than replacing interviewer judgment.The approach is described as practically deployable and aimed at approximating the patient’s perspective.
- Limitations: The study is constrained by a small dataset of 107 sessions and by evaluating transcript-based interaction quality rather than a complete therapeutic-alliance construct.The authors call for larger, more diverse corpora and note that learned fusion strategies may yield further gains.
VII. CONCLUSION
The study combines transcript-based automated predictions with post-interview interviewer ratings to estimate patients’ subjective experience in psychiatric clinical interviews. A simple average consistently outperformed either source alone, supporting a deployable framework that enhances rather than replaces post-clinical assessments.
- VII. CONCLUSION: The study evaluates five standard models on 107 interview transcripts from the German MePheSTO corpus.The analysis concerns transcript-based representations of psychiatric interviews.
- VII. CONCLUSION: A simple average of automated predictions and interviewer ratings consistently outperformed either source alone.The authors interpret this advantage as evidence that the two sources provide complementary predictive information.
- VII. CONCLUSION: The integration strategy is presented as deployable because automated language analysis can enhance, rather than replace, post-clinical assessments.The framework is based on transcript analysis and averaging.
- Study Context: The study used confidential psychiatric data collected with informed consent and ethics approval.The data’s sensitivity is linked to privacy risks.