PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCounsel-8: A Benchmark Suite for Longitudinal Depression Tracking from Multi-Session Counseling Dialogues

Jiayi Li, Zhaomin Wu, Bingsheng He

arXiv:2609.03507v1cs.LGcs.AI

TL;DR

Longitudinal depression tracking lacks multi-session counseling data with standardized session-level labels. LongCounsel-8 constructs three empirically grounded datasets through profile-based simulation and indirect behavioral realization, totaling 7,749 five-session trajectories. Validation supports label fidelity, while experiments show that magnitude and direction differ, worsening trajectories are harder, and more history may reduce trend accuracy.

  • Problem

    Existing resources usually provide either multi-session counseling without standardized depression labels or labeled interviews without longitudinal counseling dialogue.

  • Method

    LongCounsel-8 combines profile-grounded longitudinal simulation, empirical depression trajectories and symptom compositions, and indirect behavioral realization across three generated datasets.

  • Results

    Lower change MAE does not guarantee higher direction accuracy: EnsemBERT records 2.606 change MAE and 0.479 direction accuracy, while Lau et al. records 3.059 and 0.833, respectively.

  • Takeaways & Limitations

    Longitudinal evaluation should report magnitude and direction together and support direction-aware, worsening-sensitive modeling with selected longitudinal context.

  • Takeaways & Limitations

    Severe sessions are less calibrated and biased toward greater severity; their self-report MAE is 3.382 for LC8-QWEN, 1.331 for LC8-LUNA, and 1.558 for LC8-GPT-5.4-MINI.

Abstract

from arXiv · show

Tracking depression from multi-session counseling dialogues requires estimating both current symptom severity and how it changes across sessions. Yet progress on this task is constrained by the scarcity of longitudinal counseling data with standardized session-level depression labels. Existing resources typically provide either multi-session conversations without depression labels or labeled interviews in a single session. Building such a benchmark poses three challenges: maintaining longitudinal consistency and diversity, grounding symptom progression in empirical patterns, and expressing controlled depression states naturally without exposing target labels. To address these challenges, we introduce LongCounsel-8, a benchmark suite of three independently generated datasets totaling 7,749 five-session counseling trajectories, grounded in real-world client profiles, depression trajectories, symptom compositions, and counseling patterns. We combine profile-grounded simulation, empirically informed state construction, and indirect behavioral realization to address these challenges. Across the benchmark, simulated self-reports closely recover the controlled states, supporting label fidelity. Experiments on existing depression tracking methods reveal three key findings: (1) lower single-session score error does not guarantee accurate identification of trend, i.e., improvement or worsening; (2) existing methods are consistently less reliable on worsening trajectories; and (3) additional session history may reduce the accuracy of trend prediction. Together, these findings establish LongCounsel-8 as a foundation for advancing depression assessment from static, single-session prediction toward reliable longitudinal tracking of mental-health change.

1 Introduction

Longitudinal depression tracking needs multi-session counseling data with standardized session-level labels, but existing resources are incomplete. LongCounsel-8 addresses this gap through empirically grounded, profile-consistent simulation and reports distinct weaknesses in current tracking methods.

  • Existing resources generally lack standardized depression labels for every counseling session or omit aligned counseling dialogues from longitudinal measurements.
  • Benchmark construction must preserve longitudinal identity and diversity, reflect empirical symptom and counseling patterns, and express controlled states naturally without exposing labels.
  • LongCounsel-8 combines profile-grounded simulation, empirically informed depression-state construction, and indirect behavioral realization with state recovery.
  • The suite contains 7,749 five-session trajectories across three independently generated datasets and validates state fidelity, counseling-language properties, and benchmark integrity.
  • Experiments show that lower score error does not ensure correct trend identification, worsening trajectories are less reliable, and additional history can reduce trend accuracy.

2 Related Work

Prior dialogue resources cover counseling, repeated sessions, or depression supervision separately, while longitudinal health studies lack aligned counseling conversations. LongCounsel-8 combines these properties with empirical grounding and large-scale generation.

  • Counseling dialogue resources: DAIC-WOZ provides PHQ-8-supervised dialogue from one assessment interview per participant, while REALCBT provides public CBT-session transcripts.
  • Counseling dialogue resources: KokoroChat, PsyDial, CACTUS, MIRROR, TheraPhase, MusPsy, and PsychEval provide complementary role-played, reconstructed, synthetic, or multi-session counseling data.
  • Benchmark positioning: Among compared resources, LongCounsel-8 combines counseling dialogue, repeated sessions, standardized depression supervision, empirical grounding, and large-scale generation.
  • Empirical state construction: PSYCHE-D supplies five-visit depression total-score trajectories, while NHANES supplies complete item-response vectors for empirically observed symptom compositions.
  • Transcript-based assessment: Transcript-based assessment methods include supervised encoders, symptom-oriented predictors, structured LLM extraction, questionnaire completion, in-context learning, and instruction tuning.

3 Design and Construction

LongCounsel-8 constructs five-session client trajectories by combining stable profiles, treatment plans, empirical depression states, private behavioral cues, and counseling-form controls. It releases both observable self-report labels and controlled evaluation states.

  • Overview: Profile information defines stable client context, treatment plans organize visits, longitudinal totals and symptom vectors define change, and behavioral constraints connect states to dialogue.
  • Grounding sources: Each trajectory follows one client for five visits, using PsychEval profiles, PSYCHE-D trajectories, and NHANES response vectors to define identity and symptom states.
  • From symptom scores to dialogue: Symptom-score cells become private cues describing plausible lived experiences without PHQ items, response options, or numeric scores.
  • From symptom scores to dialogue: Counseling form is controlled through REALCBT-derived turn cards specifying sentence count, length, question form, and optional discourse openers.
  • Labels and evaluation: Post-session PHQ-8 self-reports provide training supervision, while the hidden controlled symptom state remains the evaluation target.
  • Dataset composition and splits: Each trajectory contains five visits with 20 counselor-client turns, and datasets use trajectory-level 7:1:2 train/validation/test splits.
  • Dataset composition and splits: Trajectories are classified as improving, worsening, fluctuating, or stable using a five-point endpoint threshold tied to clinically interpretable large change.
  • Quality control: Construction controls do not guarantee generated-output quality, so the benchmark evaluates state fidelity, counseling plausibility, safety, and resistance to shortcuts.

4 Dataset Validation

Validation tests whether the generated dialogues preserve intended symptom states, resemble counseling language, and contain transcript information beyond metadata. Results support improved cue fidelity, strong self-report recovery, and transcript-based signal beyond controls.

  • Cue pruning: Cue pruning raises item-level exact match from 0.77 to 0.85, full-profile exact match from 0.20 to 0.34, and total-score exact match from 0.25 to 0.36.
  • Counseling-language plausibility: LC8 datasets are overall closer to REALCBT in language and behavior than CACTUS and MIRROR under corpus- and snippet-level audits.
  • Self-report label fidelity: Self-report total-score MAE ranges from 0.294 to 0.541, while consecutive-change MAE ranges from 0.311 to 0.626 across datasets.
  • Transcript-label matching: No profile-conditioned metadata control improves on the global median, whereas every transcript method achieves lower current-score MAE, with the best reducing it from 3.903 to 2.594.

5 Depression Detection Methods Evaluation

The evaluation shows that depression-tracking methods differ in their ability to estimate change magnitude, identify direction, handle trajectory types, and use session history. Lower average errors do not consistently translate into better trend identification, especially for worsening trajectories.

  • Change magnitude and direction: EnsemBERT has the lowest change MAE at 2.606 but direction accuracy of 0.479, whereas Lau et al. with Qwen3 embeddings has direction accuracy of 0.833 with change MAE of 3.059.The rank reversal separates average transition-error magnitude from correct identification of improvement versus worsening.
  • Change magnitude and direction: The same reversal appears in the profile-disjoint LC8-LUNA subset, where EnsemBERT reaches change MAE 2.480 and direction accuracy 0.325, while Lau et al. reaches 3.069 and 0.828.The replicated pattern supports reporting magnitude and direction metrics together.
  • Unbalanced performance across trajectory types: Current-score MAE is lowest on stable trajectories at 2.22–3.04 and highest on worsening trajectories at 3.13–4.04 across every method.The stable-to-worsening gap ranges from 0.67 to 1.82 points, while improving and fluctuating trajectories differ by at most 0.18.
  • Using earlier sessions: Adding all available history reduces AIDA current-score MAE by 0.089 and change MAE by 0.183, but increases large-change direction error from 0.258 to 0.317.The MAE improvements and direction deterioration occur in the same method and evaluation set.
  • Using earlier sessions: History effects vary by method: complete history reduces LMIQ current-score MAE by 0.245 and change MAE by 0.265, while AIDA change MAE falls by 0.359.Direction estimates remain interval-overlapping across context conditions, so added history does not provide a uniform benefit.

6 Discussion

The benchmark findings motivate direction-aware tracking, trajectory-balanced evaluation, and dedicated designs for selecting and representing earlier-session evidence. These priorities follow from distinct magnitude-versus-direction abilities, larger errors on worsening courses, and mixed history effects.

  • Direction-aware tracking: Change magnitude and direction should be evaluated together because they require distinct capabilities.Future systems can model signed score changes or classify adjacent sessions as improving, stable, or worsening.
  • Fair methods across different trends: Worsening trajectories should be treated as a distinct modeling target because methods show their largest errors on deterioration courses.Suggested approaches include trajectory-balanced training, separate representations or losses, and trajectory-stratified accuracy and calibration.
  • History usage needs more dedicated design: Earlier-session context needs dedicated selection and representation mechanisms because its usefulness depends on how prior evidence is used.Candidate designs include retrieval, learned memory, change-point representations, and explicit trajectory models.

7 Conclusion

LONGCOUNSEL-8 provides a large benchmark for longitudinal depression tracking by combining repeated counseling dialogues with session-level PHQ supervision and controlled trajectories. Its construction and evaluation support studying direction-aware, worsening-sensitive, and history-aware assessment.

  • Conclusion: LONGCOUNSEL-8 contains 7,749 five-session counseling trajectories with controlled session-level PHQ-8 states.The benchmark bridges labeled single-session interviews and multi-session counseling corpora.
  • Conclusion: Across three independently generated datasets, validation supports controlled-state fidelity, counseling-language plausibility, and benchmark integrity.The suite is designed to evaluate current status, consecutive change, change direction, trajectory type, and history use.
  • Expanded Survey of Counseling Dialogue Resources: The resource survey distinguishes repeated sessions linked to one client or persona from a single long dialogue and requires PHQ targets aligned with each session for session-level PHQ supervision.These criteria define the scope of the expanded counseling-resource inventory.
  • Expanded Survey of Counseling Dialogue Resources: Existing human-authored and source-derived resources provide valuable dialogue structure but generally lack standardized depression severity for every encounter.DAIC-WOZ provides PHQ supervision but uses one semi-structured assessment interview per participant rather than a counseling trajectory.
  • Construction and implementation: The generation pipeline uses profile, treatment-plan, trajectory, symptom-vector, behavioral-cue, and turn-form components to connect controlled states with visible dialogue.Prompt contracts, memory handling, validation, and post-session self-reports support the released construction process.

B.2 Released data schema

The released benchmark organizes trajectories and visible transcripts under run-level contracts while excluding runtime hidden-state artifacts. Its metadata-only audit finds that profile identity and session position do not improve current-score recovery.

  • Released data schema: Each trajectory is organized under a run_id, with no separate trajectory_id field or formal JSON Schema.The release contract is defined by the writer and loader, and Table 6 summarizes the three core files.
  • Released data schema: Released transcripts contain only speaker-tagged visible text, while profiles, cues, memories, and hidden-state fields remain internal.The client-side emotion response is not written to the release transcript.
  • Generation settings: Each run contains five sessions targeting 20 counselor–client exchanges, with dataset-specific sampling and output-cap settings.Qwen self-report and summary calls use temperature 0; GPT dataset calls omit temperature.
  • Filtering and splits: Filtering precedes split construction, and completed runs require parseable transcripts plus ground truth and self-report samples for every expected session.Incomplete bundles are resumed or regenerated rather than represented by null-valued failure records.
  • Filtering and splits: Regular LC8-QWEN and LC8-LUNA manifests allow profile overlap, whereas LC8-GPT-5.4-MINI is profile-disjoint with one trajectory per profile.Manifests explicitly store run IDs and split assignments.
  • Metadata-only audit: None of the profile-conditioned controls improves on the global median for current-score recovery across the reported datasets.The profile-by-session median performs worse than the same-profile median in both overlapping-profile datasets.

C.2 Comparison with transcript-based methods

The comparison tests whether transcript content contributes severity information beyond metadata-only priors on a shared frozen split. Every transcript-based method improves current-score MAE over the selected metadata control, with Lau et al. using Qwen3 embeddings performing best.

  • Comparison design: The transcript comparison evaluates five reproducible depression-assessment methods against metadata-only priors on a shared target and split.The protocol isolates information contributed by dialogue content beyond profile and session signals.
  • Results: Every transcript-based method achieves lower current-score MAE than the validation-selected metadata-only control.All methods use the frozen LC8-QWEN split and simulated self-report labels for fitting and selection.
  • Results: 34% reduction: Lau et al. with Qwen3 embeddings lowers current-score MAE from 3.903 to 2.594.This is the best configuration in the comparison.
  • Interpretation: The results indicate that transcripts preserve aggregate severity information beyond the tested profile and session signals.This conclusion combines the metadata audit with the transcript comparison.

D Experiment Settings

Experiments use frozen splits, multiple training seeds, defined longitudinal metrics, and transcript inputs that vary by available history. The settings also specify reconstruction environments, bootstrap procedures, label-fidelity diagnostics, and multidimensional snippet audits.

  • Reproducibility: The reconstruction snapshot records Linux, eight NVIDIA A100-SXM4-80GB GPUs, Python 3.12.13, PyTorch 2.10.0, CUDA 12.8, and Transformers 5.5.1.It is a post-experiment reconstruction environment rather than a per-job historical capture.
  • Splits and seeds: The full LC8-QWEN comparison reports five training seeds, while the profile-disjoint LC8-LUNA check uses frozen 258/37/74 train, validation, and test trajectory splits.The LC8-LUNA split contains 166/27/51 profiles with zero cross-split overlap.
  • Transcript inputs: Current-session inputs and all-history inputs are defined explicitly, with no history condition receiving past PHQ labels.The all-history condition concatenates sessions through the current session using session headers and a current-session marker.
  • Methods: AIDA extracts 23 structured features and refits its linear head separately for history windows k = 0, 1, 2, and all available previous sessions.With five sessions and evaluation on sessions 2–5, k=4 coincides with all available previous sessions.
  • Longitudinal metrics: Current-score MAE measures total-score error, change MAE measures adjacent-transition error, and direction accuracy evaluates signs only for transitions with |∆y| ≥5.Predictions remain floating point and are not rounded or clipped before scoring.
  • Trajectory stratification: Trajectory types are assigned by an ordered rule: endpoint changes classify improving or worsening, internal range classifies fluctuating, and remaining cases are stable.The archived code does not implement predicted-versus-true trajectory-type agreement.
  • Validation: Label-fidelity audits report item exact match, item MAE, total-score bias, latent–self-report correlation, severity bands, and screening agreement at total ≥10.Dataset-level intervals use 2,000 trajectory resamples.
  • Validation: Multidimensional snippet outcomes use cluster-robust OLS, corrected Hedges g, and TOST equivalence with standardized margin |d| < 0.30.Holm correction is applied across the two synthetic comparisons within each outcome family.

E Limitations

The benchmark’s empirical grounding combines separate profile and depression-trajectory resources, and its audits focus on aggregate severity and conversational plausibility. Important boundaries include synthetic rather than natural counseling evaluation and weaker self-report calibration in severe sessions.

  • Scope of empirical grounding: Profile–state compatibility is evaluated only at the aggregate level because profiles and longitudinal PHQ-8 trajectories come from separate empirical resources.Jointly collected longitudinal counseling data or expert annotations could extend evaluation to item-level symptom expression.
  • Scope across generators and clinical settings: Evaluation on natural longitudinal counseling remains an important next step beyond the three generated datasets.Future studies can connect the protocol to independently established clinical states.
  • Scope across generators and clinical settings: Counseling sessions may express only a subset of the PHQ-8 state, limiting direct observation of every symptom in a conversation.The same sparsity appears in DAIC-WOZ interviews for appetite, concentration, and psychomotor symptoms.
  • Label-fidelity boundary: Severe sessions have less calibrated and less accurate self-reports, with self-report MAE of 3.382 for LC8-QWEN, 1.331 for LC8-LUNA, and 1.558 for LC8-GPT-5.4-MINI.In LC8-QWEN, severe sessions comprise 7.6% of sessions and every error shifts self-report toward greater severity.

F Asset Licenses and Data Statement

LONGCOUNSEL-8 will be released under CC BY-NC 4.0 with its datasets, labels, manifests, and generation configuration included. The data statement limits the benchmark to research use and excludes clinical decision-making or claims about real individuals.

  • LONGCOUNSEL-8 will be released under the Creative Commons Attribution-NonCommercial 4.0 International license.
  • The release includes all three datasets, both labels, split manifests, and generation configuration.
  • LONGCOUNSEL-8 is a research dataset rather than a clinical product, and model predictions must not guide diagnosis, triage, treatment, or care decisions.
  • Synthetic records must not be treated as records of real individuals or used to support claims about them.
Loading 2609.03507v1…