PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Reflection: Affirmation as a Promising Behavioral Marker Associated with Quality in Text-Based Counseling

Michimasa Inaba

arXiv:2608.26689v1cs.CL

TL;DR

The paper asks which counselor behaviors are associated with higher quality in text-based counseling, where evidence beyond Reflection-focused frameworks is limited. It analyzes newly annotated KokoroChat sessions with complementary quality indicators and tests transfer to ESConv, finding Affirmation more consistently associated with quality than Reflection and observable to some extent across datasets.

  • Problem

    Evidence is limited on how a broad range of counselor strategies comparatively relates to session-level quality in text-based counseling, despite the importance of such evidence for counselor training and AI-support design.

  • Method

    The study conducts a multi-layered analysis of newly annotated KokoroChat sessions using counselor strategies, client distress, client reviews, and cross-dataset transfer to ESConv.

  • Results

    Affirmation is more consistently associated with the study’s quality indicators than Reflection, with a KokoroChat-trained model showing ρ = −0.072, p = 0.014 on ESConv ∆emotion.

  • Takeaways & Limitations

    The findings provide empirical implications for counselor training and emotional support system design, while suggesting that the Affirmation-centered quality signal can be observed to some extent beyond KokoroChat.

  • Takeaways & Limitations

    The transfer correlation is significant but small, so strong external validity and complete operational equivalence across heterogeneous datasets cannot be assumed.

Abstract

from arXiv · show

While AI-assisted text-based counseling is gaining attention, it remains empirically unclear which counselor behaviors are associated with higher dialogue quality. Existing research often focuses heavily on Reflection, borrowing frameworks from Motivational Interviewing. To address this gap, we conduct a multi-layered analysis using KokoroChat, a large-scale Japanese text counseling dataset conducted by professional counselors and trainees, newly annotated with counselor strategy tags and client distress levels. Our results show that, under the quality indicators used in this study, Affirmation is more consistently associated with session quality than Reflection among the analyzed strategies. Cross-dataset transfer experiments further suggest that this quality signal can be observed to some extent on ESConv, an English dataset with non-expert supporters. These findings provide empirical implications for counselor training and emotional support system design. We release the additional KokoroChat annotations and experimental source code at https://github.com/UEC-InabaLab/BeyondReflection.

1 Introduction

The paper addresses limited comparative evidence about counselor behaviors associated with quality in text-based counseling. It examines these associations using newly annotated KokoroChat data and cross-dataset transfer, finding Affirmation more consistently associated with quality than Reflection.

  • Research gap: Empirical evidence remains limited on how diverse counselor strategies relate to session-level quality indicators in text-based counseling.The authors frame this evidence as relevant to both human counselor training and AI-assisted support-system design.
  • Related work: Existing work often uses Motivational Interviewing categories, with Reflection receiving particular attention in behavior classification, generation, and detection.The paper notes that MI primarily targets behavior change, whereas emotional support counseling targets distress reduction and emotional stability.
  • Related work: Reflection and Restatement have also been analyzed as important strategies in emotional support and online mental-health dialogue research.Mixed-methods work additionally links active listening, reflective restatements, and exploration opportunities with perceived empathy, while finding human–computational assessment discrepancies.
  • Approach: The study uses KokoroChat counselor evaluations, new counselor-strategy and client-distress annotations, and transfer validation on ESConv to examine behavioral associations with quality.The cross-dataset analysis uses commonly definable behavioral features to test whether findings are specific to KokoroChat.
  • Contributions: Across the analyzed strategies, Affirmation is more consistently associated with the study’s quality indicators than Reflection, and this signal is observable to some extent beyond KokoroChat.The study adds annotations for 6,589 sessions and releases the annotations and experimental code.

2 Data and Annotation

The study analyzes 6,589 Japanese role-play counseling sessions using newly added strategy and distress annotations. Counselor utterances receive mutually exclusive strategy tags, while client utterances receive four-level distress labels validated against human annotations.

  • Dataset: KokoroChat contains 6,589 one-on-one, 60-minute Japanese text-counseling sessions conducted by professional counselors or trainees.Client-role speakers provide 20-item evaluation scores ranging from 0 to 100.
  • Counselor strategies: Each counselor utterance is assigned one of 11 mutually exclusive strategy tags in the new annotation scheme.The scheme adopts and adapts categories from ESConv and Anno-MI, including distinctions among question types and reflection types.
  • Counselor strategies: Affirmation evaluates client strengths, efforts, motivations, abilities, or actions, whereas Reflection verbalizes emotion or underlying meaning and Paraphrase restates factual content.Emotion validation alone is classified as Reflection rather than Affirmation.
  • Distress levels: Client utterances are annotated on a 4-point distress scale from 0, no distress, to 3, severe distress.Consecutive client utterances are treated as one block, and annotations are produced automatically.
  • Validation: Human validation showed substantial agreement for both counselor strategies and distress levels, including worker–worker and Gemini–worker comparisons.For example, counselor-strategy worker agreement was Cohen’s κ = 0.674, while distress-level worker agreement was Quadratic Weighted κ = 0.677.

3 Analysis of Counselor Strategies and Session Quality

Session quality is assessed through client-rated score tiers and changes in client distress, capturing subjective evaluation and state transition. Affirmation is the most consistently associated strategy across these analyses, while the observational design limits causal interpretation.

  • Quality indicators: The analysis combines three client-review score tiers with session-level distress change, where more negative ∆distress indicates greater distress reduction.The two indicators represent subjective evaluation and client-state transition.
  • Quality indicators: The two quality indicators are significantly negatively correlated, with monotonic mean ∆distress across score tiers, while capturing related but non-identical aspects of quality.This alignment supports using both indicators in complementary analyses.
  • Tier-based analysis: Affirmation accounted for 9.9% of counselor utterances in the high tier versus 6.5% in the low tier, a 3.5-percentage-point absolute difference.High-tier sessions also showed higher Affirmation and lower question usage in Bonferroni-corrected tier comparisons.
  • Distress-based analysis: Affirmation was the strategy most consistently associated with distress alleviation, showing a significant negative correlation with distress change.The authors hypothesize that explicit acceptance and validation may foster security and ease of speaking when nonverbal cues are limited.
  • Robustness and scope: Temporal robustness checks and mixed-effects/Mundlak analyses did not eliminate the Affirmation association, but the results remain correlational.The absolute correlation is modest, so Affirmation alone should not be interpreted as strongly explaining session quality.

4 Cross-Dataset Transfer

The cross-dataset experiment tests whether behavioral patterns associated with high quality in Japanese KokoroChat transfer to English ESConv. The transferred signal is statistically detectable but small amid differences in language, culture, expertise, and evaluation metrics.

  • Objective: The transfer experiment asks whether a KokoroChat quality model yields meaningful predictive signals on ESConv, which differs in language, culture, and supporter expertise.KokoroChat uses Japanese trained counselors, whereas ESConv uses English non-expert supporters.
  • Experimental setup: The model is trained on high-versus-low KokoroChat review-score tiers using 4,390 sessions after excluding the middle tier.The setup prioritizes sessions with the clearest quality contrast.
  • Experimental setup: The transfer features use usage rates for Affirmation, Reflection, Question, Suggest, and Other, mapped to semantically aligned ESConv categories.Question combines OpenQuestion and ClosedQuestion in KokoroChat.
  • Results: The KokoroChat-trained model’s predicted high-quality probability correlated negatively with ESConv ∆emotion, ρ = −0.072, p = 0.014.∆emotion is the difference between final and initial emotion intensity; the direction suggests greater emotional improvement for behavior patterns classified as high quality in KokoroChat.
  • Results: Affirmation had the largest positive transfer-model coefficient, +0.536, while Question had a large negative coefficient, −0.421.Suggest and Reflection were positive but smaller, whereas Other was negative.
  • Limitations: The transfer association is modest and does not establish strong external validity across heterogeneous datasets.Differences in language, dialogue setting, annotation procedures, and evaluation metrics prevent assuming complete operational equivalence.

5 Conclusion

The study finds that Affirmation is more strongly and consistently associated with session quality than Reflection under its evaluation setting. Transfer to ESConv suggests this Affirmation-related quality signal persists to some extent across languages and expertise levels.

  • Affirmation showed a stronger and more consistent association with quality indicators than Reflection in the analyzed KokoroChat sessions.The conclusion is limited to the study’s present evaluation setting.
  • Cross-dataset transfer to ESConv suggested that the quality signal associated with Affirmation can be observed to some extent across languages and expertise levels.
  • The findings provide empirical implications for counselor training and emotional support system design.

Limitations

The study’s conclusions are constrained by potential circularity in KokoroChat quality measures, weak external validity, residual counselor-level confounding, noisy automatic annotations, observational design, and modest correlations.

  • KokoroChat’s client-review tiers are not fully independent of other session-derived metrics, creating potential partial circularity in the internal analysis.Transfer validation using a different metric on ESConv provides some independent support for the main conclusion.
  • The significant transfer correlation is small, so strong external validity and high-accuracy application to heterogeneous datasets cannot yet be claimed.
  • Unobserved counselor competencies, including timing, empathy, and rapport-building, may contribute to the observed Affirmation–quality association.Modeling counselor identity does not fully isolate all counselor-level characteristics.
  • LLM-generated annotations contain inherent noise and potential biases despite substantial agreement with human annotators and additional error analysis.
  • The observational design does not support causal inference, and temporal analyses do not rule out all confounding.The study therefore treats Affirmation as an associated behavioral marker rather than a causal determinant of improvement.
  • The observed correlations are modest, indicating that macro-level strategy frequencies explain only a small portion of session-quality variance.Reported examples include ρ = −0.201 for Affirmation usage and ∆distress and ρ = −0.072 for cross-dataset transfer.

Ethical Considerations

The study uses publicly available role-play counseling datasets under their terms of use and emphasizes ethical safeguards for data, annotation, and deployment. Its behavioral markers are intended primarily for human counselor training and human-in-the-loop systems, not fully autonomous mental-health care.

  • KokoroChat and ESConv are publicly available datasets used according to their respective terms of use and ethical protocols.
  • KokoroChat contains role-play sessions with professional counselors and trainees, without real patients or personally identifiable information from actual clients.
  • CrowdWorks annotators were compensated at an hourly rate exceeding Tokyo’s minimum wage, approximately 7 USD per hour.
  • Because harmful autonomous responses could exacerbate client distress, the identified behavioral markers are intended primarily for human counselor training and human-in-the-loop AI support.

D Tag-Level Error Analysis of Automatic Strategy Annotations

The tag-level analysis finds substantial overall agreement between Gemini and human annotators, but errors vary by category and cluster around semantically adjacent strategies. Affirmation is rarely missed but sometimes over-assigned to neighboring supportive responses.

  • Cohen’s κ was 0.710 and 0.687 between the LLM and two human annotators, compared with 0.674 between the human annotators.LLM–human agreement was comparable to or slightly higher than human–human agreement.
  • Tag-level errors were unevenly distributed, with Other receiving the lowest F1 and question sub-types showing moderate performance.
  • OpenQuestion and ClosedQuestion were frequently confused, while Paraphrase and Reflection also showed ambiguity because of their conceptual proximity.
  • The LLM achieved perfect recall for Affirmation against both human annotators.Every utterance labeled Affirmation by either human annotator was also labeled Affirmation by the LLM.
  • Affirmation precision was lower because false positives mainly came from human-labeled Reflection, Paraphrase, or Thanking utterances.Thus, the estimated Affirmation rate may include some neighboring supportive strategies.
  • Downstream findings should be interpreted as applying to automatically identified Affirmation-like behavior rather than exclusively to human-defined Affirmation.

E Relationship Between Client-Rated Scores and Distress Change

Client-rated scores and distress change aligned in the expected direction but captured complementary aspects of session quality rather than interchangeable outcomes.

  • ρ = −0.269, p < .001, n = 6,589: Client-rated review scores were significantly negatively correlated with ∆distress.Because lower ∆distress indicates greater distress reduction, higher-rated sessions tended to involve greater distress reduction.
  • High-tier sessions showed the largest distress reduction, while low-tier sessions showed an increase in distress.Mean ∆distress was −0.204 for high-tier, −0.053 for mid-tier, and 0.142 for low-tier sessions.
  • The two indicators were treated as complementary because their modest correlation suggests they are not interchangeable.Client-rated scores reflect broader subjective quality, whereas ∆distress captures short-term emotional change within the session.
  • Table 5 reports strategy usage rates and utterance counts across client-rated score tiers, including the high-minus-low tier difference.Usage rates were calculated over all counselor utterances within each tier.

G Temporal Robustness Checks for Affirmation

Temporal analyses reduced the plausibility that Affirmation-quality associations were caused simply by counselors responding after clients had improved, while retaining a correlational interpretation.

  • Early-session Affirmation was significantly associated with both distress reduction and overall client-rated scores.It was negatively correlated with ∆distress (ρ = −0.101, p < .001) and positively correlated with client-rated scores (ρ = 0.126, p < .001).
  • Early-session associations indicate that the Affirmation-quality relationship was present before the late phase of sessions.Late-session Affirmation showed stronger associations, but the finding was not simply attributable to counselors increasing Affirmation after client improvement.
  • ρ = 0.008, p = .499: Early distress did not significantly predict later Affirmation usage in a simple correlation analysis.This provided no clear support for the explanation that lower early distress leads to greater later Affirmation use.
  • OR = 1.068, 95% CI [1.050, 1.086], p < .001: Higher current distress predicted greater likelihood of Affirmation in the next counselor turn.A session-clustered GEE produced the same conclusion (OR = 1.105, p < .001).
  • Affirmation-quality associations varied by session phase: distress associations appeared in middle and late phases, while score associations appeared across all phases.The strongest distress association occurred in the late phase.
  • The analyses did not establish causality or rule out all confounding, so Affirmation was interpreted as a temporally robust behavioral marker rather than a causal determinant.

H Counselor-Level Random Effects and Mundlak Decomposition

Counselor-level analyses found that Affirmation associations persisted after accounting for stable counselor differences, although the observational design still limits causal interpretation.

  • Counselor-level random effects: ICCs were 0.028 for ∆distress and 0.110 for client-rated score, indicating small distress-change clustering but nonnegligible counselor heterogeneity in subjective ratings.
  • Counselor-level random effects: After including counselor-level random intercepts, Affirmation remained associated with ∆distress and client-rated score.The coefficients were −1.213 for ∆distress and 85.517 for client-rated score, both p < .001.
  • Mundlak decomposition: Within-counselor Affirmation variation remained significant for both outcomes: coefficient = −1.277 for ∆distress and 88.035 for client-rated score, both p < .001.Thus, for the same counselor, sessions with higher Affirmation usage tended to show better outcomes.
  • Mundlak decomposition: Counselors with higher mean Affirmation rates tended to show greater distress reduction and higher average client ratings.Between-counselor coefficients were −0.773 for ∆distress and +51.828 for client-rated score.
  • Interpretation: The association involving Affirmation was unlikely to be solely attributable to fixed counselor differences, but between-counselor associations may reflect other stable competencies.These analyses remained correlational and did not establish a causal effect of Affirmation.
Loading 2608.26689v1…