PaperlayerView exact revision on arXiv

Source-linked AI summary

Candidate Generation and Definition-Guided Verification for Sentence-Level Depression Symptom Recognition

Weiming Li, Catarina Barata, Miguel Constante, Joao Sanches

arXiv:2609.01833v1cs.CL

TL;DR

Sentence-level depression symptom recognition must distinguish symptom categories from whether textual evidence actually supports them, especially when expressions are ambiguous or context-dependent. The paper proposes a two-stage encoder–LLM pipeline that generates a candidate and verifies it against a candidate-specific definition. Across evaluated methods, the pipeline achieves the strongest reported overall performance, while rare categories and diagnostic scope remain limitations.

  • Problem

    Existing depression analyses often operate at post or user level, while sentence-level recognition must resolve overlapping, contextual, and long-tailed symptom expressions.

  • Method

    A contrastively fine-tuned MentalSBERT-S encoder selects one symptom candidate per sentence, then a fine-tuned DeepSeek verifier checks Present or Absent against a candidate-specific DSM-5-informed definition using sentence and post context.

  • Results

    0.731 ± 0.005 accuracy, 0.690 ± 0.011 Macro F1, and 0.715 ± 0.010 Weighted F1 were achieved across three seeds, outperforming all evaluated baselines.

  • Takeaways & Limitations

    The evidence supports decomposing recognition into candidate generation and definition-grounded verification for sentence-anchored symptom hypotheses and rationales, not diagnosis.

  • Takeaways & Limitations

    Rare categories remain difficult, and the small, long-tailed dataset is not clinically representative; outputs do not establish duration, impairment, or episode-level diagnostic criteria.

Abstract

from arXiv · show

Sentence-level recognition of depression symptoms is challenging because similar expressions can differ in symptom relevance, and language-model inference is insufficiently grounded in diagnostic definitions. This study proposes a two-stage framework separating symptom-candidate generation from definition-grounded verification. A contrastively fine-tuned sentence encoder generates a symptom candidate per sentence, and a fine-tuned language model verifies whether the candidate is present or absent using the sentence, its context, and a candidate-specific diagnostic definition, checking its judgment against that definition before answering. Evaluated against encoder, inference-based, medical, and general LLM baselines and a matched single-stage supervised classifier, the proposed pipeline attains the best accuracy and F1 scores of all methods, with rationales matching expert-authored annotations. A preliminary clinical audit indicates moderate alignment with diagnostic definitions, with explanation quality strongly dependent on prediction correctness. The results support decomposing symptom recognition into candidate generation and definition-grounded verification, though performance remains limited for rare categories.

I. INTRODUCTION

Sentence-level depression symptom recognition addresses a gap left by post- and user-level screening, where symptom-specific sentence evidence is not localized. The proposed framework combines contrastive candidate generation with definition-grounded verification to handle ambiguous, contextual symptom expressions.

  • Post- and user-level depression analysis typically produces aggregate status or severity outcomes rather than identifying symptom-bearing sentences.
  • Sentence-level recognition is difficult because symptom expressions can be implicit, overlapping, negated, temporally qualified, or context-dependent, while categories are long-tailed.
  • Strong encoders can struggle near overlapping symptom boundaries, whereas LLMs add cost and remain vulnerable to inconsistency, bias, prompt sensitivity, and imperfect clinician agreement.
  • The framework generates a nine-class symptom candidate with MentalSBERT-S before a fine-tuned DeepSeek model verifies Present or Absent using context and a candidate-specific definition.
  • The study evaluates the pipeline against supervised, NLI-style, medical-LLM, direct-LLM, and matched single-stage alternatives using multi-seed and clinical-rationale analyses.
  • Definition-conditioned inference supplies a fixed, transparent criterion for checking whether a sentence and its local context support a proposed symptom, but remains an annotation-level consistency check.

III. METHOD

The method separates symptom-category localization from symptom-presence verification. A sentence encoder proposes the most plausible candidate, and a verifier uses the sentence, source-post context, and a candidate-specific definition to produce a checked status and explanation.

  • The framework treats candidate selection and evidence verification as distinct decisions because category confusion and symptom validity are different error sources.
  • ReDSM-5 uses the target sentence as the analytical unit while retaining the linked source post as supporting context for context-dependent verification.
  • Stage 1: Stage 1 applies MentalSBERT-S to sentence x, produces independent symptom scores P(x), and selects the highest-scoring category as s(x).
  • Data format: The dataset format independently labels sentences with a DSM-5 symptom category, Present/Absent status, and human-written rationale within source-post context.
  • Stage 1: MentalSBERT-S combines multi-head binary classification with triplet-contrastive learning, using same-label sentences as positives and different-label sentences as negatives.
  • Stage 1: Independent classification heads allow overlapping symptoms to retain non-negligible scores before the selected candidate is passed to Stage 2.

B. DEFINITION-GROUNDED BINARY SYMPTOM VERIFICATION

Stage 2 verifies whether the Stage-1 symptom candidate is present or absent rather than selecting another category. It uses sentence evidence, supporting post context, and definition-guided self-checking for a structured annotation-level judgment.

  • Stage 2 formulates verification as binary classification, assigning Present or Absent to the symptom candidate produced by Stage 1.
  • The target remains sentence x, while source post d is included only to resolve negation, referent ambiguity, or context-dependent meaning.
  • A separately LoRA-adapted DeepSeek decoder performs the Present/Absent task using a sequence-level training objective.
  • The verifier records an initial binary judgment and is instructed to prioritize explicit sentence evidence, use context supportively, and distinguish current from negated or historical evidence.
  • The model returns structured output without diagnosing major depressive disorder or requiring evidence for duration, severity, functional impairment, or all diagnostic criteria.

C. DEFINITION-GROUNDED SELF-CHECKING AND EXPLANATION GENERATION

The framework uses candidate-specific DSM-5-informed definitions to verify binary symptom judgments against the target sentence and context, while treating these definitions as semantic references rather than complete diagnostic criteria.

  • C. DEFINITION-GROUNDED SELF-CHECKING AND EXPLANATION GENERATION: Candidate-specific DSM-5-informed definitions guide verification of each proposed symptom.They serve as sentence-level consistency references rather than complete diagnostic assessments.
  • C. DEFINITION-GROUNDED SELF-CHECKING AND EXPLANATION GENERATION: The verifier checks its preliminary Present/Absent judgment against the target sentence, source-post context, and candidate definition.The initial judgment and consistency check are elicited within one structured verifier call.
  • C. DEFINITION-GROUNDED SELF-CHECKING AND EXPLANATION GENERATION: The dataset records target sentences, source-post context, symptom labels, candidate-specific status, and annotation rationales.ReDSM-5 supports nine-class symptom localisation and candidate-specific binary verification.
  • C. DEFINITION-GROUNDED SELF-CHECKING AND EXPLANATION GENERATION: Table 2 presents DSM-5-informed symptom categories and definitions used as semantic references for symptom-related evidence.These references do not require duration, frequency, severity, impairment, or differential-diagnosis criteria.

B. STAGE-1 RESULTS ANALYSIS

Stage 1 evaluates nine-class symptom-candidate generation on a fixed 324-sentence test partition, comparing supervised encoders, prompting baselines, generative models, ablations, and efficiency.

  • B. STAGE-1 RESULTS ANALYSIS: 0.404 accuracy, 0.353 Macro F1, and 0.368 Weighted F1 were obtained by BART-MNLI, compared with 0.716, 0.684, and 0.726 for MedGemma-4B-IT.The zero-shot MentalBERT cosine-similarity classifier reached 0.704 accuracy and 0.731 Macro F1.
  • B. STAGE-1 RESULTS ANALYSIS: 0.934 ± 0.002 accuracy, 0.898 ± 0.002 Macro F1, and 0.932 ± 0.002 Weighted F1 made MentalSBERT-S stronger numerically than MentalRoBERTa.The Macro-F1 difference had a 95% confidence interval of [−0.019, 0.065] and p = 0.334, so the advantage was not statistically conclusive.
  • B. STAGE-1 RESULTS ANALYSIS: 0.918 ± 0.010 accuracy and 0.891 ± 0.020 Macro F1 were obtained by directly fine-tuned DeepSeek-R1-Distill-Qwen-14B, slightly below MentalSBERT-S.Unlike the generative model, MentalSBERT-S requires no autoregressive generation for each sentence.
  • B. STAGE-1 RESULTS ANALYSIS: 0.923 ± 0.003 accuracy and 0.881 ± 0.024 Macro F1 were obtained by the BCE-only variant.The full BCE+triplet objective therefore provides a modest improvement at this data scale.
  • B. STAGE-1 RESULTS ANALYSIS: 0.934 ± 0.002 accuracy and 0.898 ± 0.002 Macro F1 were achieved by MentalSBERT-S with the full BCE+triplet objective.Adding contrastive learning improved accuracy by 0.011 and Macro F1 by 0.017 over the corresponding variant.
  • B. STAGE-1 RESULTS ANALYSIS: 0.477 to 0.586 was the PSYCHOMOTOR F1 improvement associated with adding the contrastive term.PSYCHOMOTOR was the lowest-support category and was often confused with FATIGUE and DEPRESSED_MOOD.
  • B. STAGE-1 RESULTS ANALYSIS: Under 0.5 seconds and under 6,000 tokens were required by MentalSBERT-S for the 324-sentence test partition.Generative baselines required between 8.3 × 10^4 and 4.3 × 10^5 total tokens.

C. STAGE-2 RESULTS ANALYSIS

Stage 2 verifies Stage-1 symptom candidates as Present or Absent using candidate-specific DSM-5-informed definitions and self-checking. The proposed verifier outperforms evaluated baselines numerically, but bootstrap tests do not establish statistically significant differences.

  • Stage-2 design: The verifier evaluates each Stage-1 candidate as Present or Absent using the target sentence, source-post context, and a candidate-specific DSM-5-informed definition.A structured inference call produces a binary judgment and checks it against the supplied definition.
  • Ablation analysis: Adding only the DSM-5 definition lowers Macro F1 to 0.630 ± 0.052, while the proposed self-check configuration recovers and slightly exceeds the direct setting.Direct fine-tuning reaches 0.766 ± 0.011 accuracy and 0.656 ± 0.019 Macro F1; pairwise bootstrap differences are not significant.
  • Baseline comparison: 0.780 accuracy and 0.862 Present F1 exceed DeepSeek RAG’s 0.741 accuracy and 0.830 Present F1.The fine-tuned verifier’s Macro-F1 difference from DeepSeek RAG is not statistically conclusive: 95% CI [−0.055, 0.082], p = 0.665.
  • Verification outcomes: 224 of 244 Present and 29 of 80 Absent instances are correctly identified, with 51 false accepts versus 20 false rejects.The false-accept pattern is consistent with lower Absent F1 and class imbalance.
  • Category variation: F1 reaches 0.952 for FATIGUE and 0.971 for SUICIDAL_THOUGHTS but falls to 0.615 for PSYCHOMOTOR and 0.690 for COGNITIVE_ISSUES.Category-level results indicate greater reliability for semantically clearer categories and difficulty for rarer categories.

D. TWO-STAGE INFERENCE RESULT ANALYSIS

The complete pipeline generates a symptom candidate with MentalSBERT-S and then verifies it with a fine-tuned DeepSeek model, mapping rejected candidates to NO_SYMPTOM. It achieves the best overall end-to-end metrics among evaluated methods, although the advantage over a matched single-stage baseline is not statistically conclusive.

  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: Stage 1 proposes one of nine symptom categories, and Stage 2 accepts or rejects that candidate; rejection produces NO_SYMPTOM.The final task therefore contains nine symptom categories plus NO_SYMPTOM without access to gold symptom labels.
  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: 3.020 ± 0.093s/sample is the pipeline latency, while DeepSeek prompting baselines exceed 3.6s/sample and use 2–5× more total tokens.Stage 1 adds under half a second in total across the 324 test sentences.
  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: 0.731 ± 0.005 accuracy, 0.690 ± 0.011 Macro F1, and 0.715 ± 0.010 Weighted F1 are the best results of any evaluated method.The matched single-stage baseline uses the same backbone, training partition, and comparable QLoRA settings.
  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: The two-stage pipeline’s Macro-F1 advantage over the matched single-stage baseline is not statistically conclusive: 95% CI [−0.053, 0.105], p = 0.498.The numerical advantage is consistent in direction across all three seeds.
  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: F1 is strongest for SUICIDAL_THOUGHTS (0.939), WORTHLESSNESS (0.889), FATIGUE (0.870), and DEPRESSED_MOOD (0.800), but weakest for PSYCHOMOTOR (0.364) and NO_SYMPTOM (0.453).PSYCHOMOTOR has only 6 test instances, while NO_SYMPTOM reflects accumulated Stage-1 and Stage-2 errors.
  • D. TWO-STAGE INFERENCE RESULT ANALYSIS: Correct verifier decisions receive average audit scores of 3.52/2.73, versus 2.11/1.90 for incorrect decisions on DSM-5 alignment and clinical usefulness.The audit used 32 explanations stratified by predicted status and correctness.

V. DISCUSSION

The study’s two-stage decomposition is the best-performing configuration, but its advantages remain trends under limited statistical power and important data, architectural, and audit constraints.

  • Overall findings: The two-stage pipeline is the best-performing configuration among all evaluated methods, including single-stage alternatives.Its numerical margin over the matched single-stage baseline is not statistically significant at the current test-set scale.
  • Stage-wise analysis: MentalSBERT-S matches the directly fine-tuned generative alternative’s accuracy without autoregressive generation, reducing candidate-generation cost.The reported comparison is described as having no measurable accuracy cost.
  • Stage-wise analysis: Adding the DSM-5 definition alone numerically reduces Stage-2 Macro F1, whereas adding self-checking recovers and slightly exceeds the direct-only level.None of the pairwise ablation differences reaches significance, so the finding is reported as a trend.
  • End-to-end comparison: The pipeline outperforms the matched single-stage baseline on accuracy, Macro F1, and Weighted F1 across all three seeds, without a significant hierarchical-bootstrap difference.A larger evaluation partition is needed to determine whether the gap is stable.
  • Trade-offs: Two-stage inference is slower than single-stage inference because it makes two model calls and generates a supporting rationale, although it remains cheaper than prompting baselines.The pipeline also consumes fewer total tokens overall than prompting-based alternatives.
  • Limitations: Rare categories such as PSYCHOMOTOR remain difficult, and an incorrect Stage-1 candidate accepted by Stage 2 becomes an unrecoverable end-to-end error.Stage 2 cannot substitute a different symptom category under the current architecture.

VI. CONCLUSION

The study combines encoder-based candidate generation with definition-grounded verification for sentence-level DSM-5 symptom annotation. Across three seeds, it reports the strongest aggregate performance among evaluated methods, while emphasizing that the system is for research annotation rather than diagnosis.

  • Framework: The framework separates identifying the most plausible symptom candidate from deciding whether available evidence is sufficient to retain it.This separation combines MentalSBERT-S candidate generation with a definition-grounded, self-checking verifier.
  • Framework: Stage 1 selects an initial candidate without autoregressive generation, while Stage 2 uses sentence, post context, and a DSM-5-informed definition for Present/Absent verification.Absent maps the sentence to NO_SYMPTOM, whereas Present retains the Stage-1 candidate.
  • Results: 0.731 ± 0.005 accuracy, 0.690 ± 0.011 Macro F1, and 0.715 ± 0.010 Weighted F1 were achieved across three seeds, outperforming all evaluated baselines.The margin over the matched single-stage DeepSeek QLoRA model was not statistically significant.
  • Results: 0.597 ± 0.003 Sentence-BERT similarity was obtained with expert rationales on jointly correct predictions.A preliminary psychiatrist audit reported population-weighted DSM-5 alignment and clinical usefulness scores of 3.15 and 2.52.
  • Limitations: The framework’s limitations include a small long-tailed dataset, ambiguous expressions, dependence on Stage-1 candidates, limited statistical power, and a preliminary clinical audit.Its outputs are intended as sentence-level research annotations and explanations, not psychiatric diagnoses.

A. SOFTWARE ENVIRONMENT

The software environment documents the reproducible model configurations, training setup, symptom definitions, and data handling used by the two-stage system.

  • Software environment: The retained fine_tuning_deepseek environment used identical package versions and model identifiers across all three random-seed runs.The exact package versions are listed in Table 16.
  • Stage 1: MentalSBERT-S uses all-mpnet-base-v2 with nine one-vs-rest sigmoid heads for symptom-candidate generation.Its principal training settings are reported in Table 17.
  • Stage 2: The Stage-2 verifier is initialized from DeepSeek-R1-Distill-Qwen-14B and adapted with 4-bit QLoRA for Present/Absent verification.Its fine-tuning configuration is summarized in Table 18.
  • Data and training: Training uses 1,163 instances for training and 280 for development without oversampling or class reweighting.The training labels are imbalanced toward Present, with 904 Present and 259 Absent instances.
  • Verifier inputs: The verifier receives one target sentence, one candidate symptom, its Present/Absent status, and a candidate-specific DSM-5-informed definition.The task-specific input/output specification is documented in Table 19.
  • Symptom definitions: The supplied definitions operationalize nine symptom categories, including anhedonia, appetite change, cognitive issues, depressed mood, fatigue, psychomotor symptoms, sleep issues, suicidal thoughts, and worthlessness.Each definition specifies the symptom meaning used for verification.

E. STAGE-2 VERIFIER PROMPT TEMPLATE

The verifier prompt structures sentence-level annotation around a candidate symptom, local context, and a DSM-5-informed definition, followed by judgment, consistency checking, and explanation.

  • Inputs and task: The verifier receives the candidate symptom, its DSM-5 guideline, the target sentence, and the source post context.These fields are represented as SYMPTOM, DSM5_GUIDELINE, TARGET_SENTENCE, and CONTEXT_POST.
  • Inputs and task: The prompt asks whether the target sentence expresses the requested symptom label as PRESENT or ABSENT.The target sentence is primary evidence, while the post context provides supporting local context.
  • Scope: The task is explicitly sentence-level annotation rather than formal diagnosis and does not require duration, severity, or recurrence.The prompt also states that brief, mild, implicit, or short expressions can be present when meaningfully expressed.
  • Decision procedure: The initial judgment is set to 1 when the sentence or local context meaningfully expresses the candidate symptom and to 0 otherwise.The prompt defines this as the first inference step.
  • Decision procedure: The model conservatively checks the initial judgment against the DSM-5 guideline and either retains or corrects it.Allowed actions include KEEP_1, KEEP_0, CORRECT_TO_1, and CORRECT_TO_0.
  • Explanation and output: The prompt requires a DSM-5-style explanation supporting the final label without mentioning annotations, gold labels, training data, or technical processing.The output is one JSON object with self_label, final_label, verify_action, and reason fields.

F. INFERENCE SUMMARY

The section presents the Stage-2 verifier’s inference settings and an example output from the proposed framework. In the example, the model identifies worthlessness as absent based on positive self-perception and life satisfaction.

  • Inference settings: TABLE 20 specifies the decoding and generation settings used by the final Stage-2 verifier.These settings were applied identically across reported test-partition evaluations and the clinician explanation audit.
  • Example output: The framework generates WORTHLESSNESS as the candidate symptom for the target sentence.The target sentence states, “I’m already happy with who I am and what I do.”
  • Example output: The supporting post context repeats the speaker’s positive self-assessment and satisfaction with identity and activities.The context situates the target sentence within a question about self-improvement.
  • Example output: The model explains that the sentence is not indicative of worthlessness because it expresses positive self-perception and life satisfaction.The explanation treats the candidate symptom as absent in this example.
Loading 2609.01833v1…