PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

Xinkai Chen

arXiv:2609.39049v2cs.CLcs.AI

TL;DR

The paper asks whether marking clinical criteria and converting their count to a severity label provides a useful alternative to direct prompting and chain-of-thought. It evaluates these approaches across two Reddit corpora, three LLMs, and two questionnaires; extraction shows no consistent frontier-model advantage, and greater ordinal agreement does not ensure fewer missed severe cases.

  • Problem

    LLM severity labels are harder to audit than criterion-by-criterion judgments, motivating evaluation of a count-based alternative against direct and chain-of-thought prompting.

    Source: §I · p. 1

  • Method

    The study compares direct prompting, chain-of-thought, and structured PHQ-9 or BDI-II criteria extraction across two Reddit corpora and three LLMs using quadratic weighted kappa.

    Source: §I · p. 1

  • Results

    Frontier models show no significant criteria-extraction advantage over chain-of-thought across the tested threshold regimes, while severe-case misses increase in nearly all comparisons as the approach moves from direct prompting to chain-of-thought to extraction.

    Source: §I · p. 1, §VIII · pp. 7–8

  • Takeaways & Limitations

    Higher ordinal agreement does not ensure better detection of severe cases, and the paper concludes that error direction should be reported alongside error size.

    Source: §I · p. 1

    DepSeverity has no depression-community source, and DepSign’s 706-post test sample limits power for its 50 SEVERE posts.

    Source: §VI · p. 7

Abstract

from arXiv · show

A large language model (LLM) can rate depression severity directly from a social media post, or mark which clinical criteria the post shows and let a simple rule turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts as severe, whether prompted directly or with chain-of-thought, while the a priori rule, which uses no labels, beats both. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On one corpus, a relabeled stress dataset, a model using that dataset's own features, some computed from the post text, is not significantly different from frontier criteria extraction under the a priori rule.

I. INTRODUCTION

The paper evaluates whether extracting clinical criteria and counting them offers a more auditable alternative to direct severity labels and chain-of-thought. It finds that apparent extraction gains depend on fitted thresholds, while severe-case misses and DepSeverity’s source-data origin qualify the benchmark results.

I. INTRODUCTION

  • The study compares three LLMs on DepSeverity and DepSign, evaluating ordinal-label agreement with quadratic weighted kappa.The corpora use four and three severity labels, respectively.Source: §I · p. 1
  • For frontier models, criteria extraction’s apparent gains depend on fitted thresholds; with a priori thresholds, no significant advantage over chain-of-thought remains.The comparison is confounded when extraction thresholds are fitted on labels but the prompting baseline is zero-shot.Source: §I · p. 1
  • The 9B model’s results differ: fitted extraction gains over chain-of-thought, but a priori thresholds remove the gain on DepSeverity and beat both prompting conditions on DepSign.Recalibrating chain-of-thought with the same labels leaves no significant gap on DepSign, consistent with a calibration effect.Source: §I · p. 1
  • Across 11 of 12 comparisons, moving from direct prompting to chain-of-thought and then extraction increased the share of severe posts assigned a lower label.Severe posts make up 7–8% of posts, and extraction’s 9B-model gain coincided with 47 of 50 severe posts missed.Source: §I · p. 1
  • DepSeverity is a relabeled copy of Dreaddit, a stress dataset whose source communities are not depression-focused.A model using Dreaddit’s own features, including text-derived word-category counts, reaches κw = 0.404, not significantly different from frontier extraction with a priori thresholds.Source: §I · pp. 1–2

II. RELATED WORK

  • Earlier work annotated posts by individual criteria, while this study evaluates extraction against severity labels and examines how criteria become a label.Prior work includes DSM-5 and PHQ-9 criteria annotations; the authors state that they know of no work measuring the effect of fitting thresholds against zero-shot prompting.Source: §II · p. 2

A. DepSeverity and its origin

DepSeverity reuses posts from Dreaddit and assigns them severity labels, whereas DepSign draws from mental-health communities. The paper describes the prompting conditions and structured criteria extraction used to compare the corpora.

A. DepSeverity and its origin

  • DepSeverity’s 3,530 deduplicated posts appear word for word in Dreaddit, making it a relabeling rather than an extension of that stress dataset.Its source communities are not depression communities, and only 3 of 56 gold-SEVERE test posts mention self-harm.Source: A. DepSeverity and its origin · p. 2

B. DepSign

  • DepSign comes from mental-health subreddits, and two domain experts labeled its posts against written guidelines with Cohen’s κ = 0.686.The test split is subsampled to 706 posts, matching DepSeverity’s test size.Source: B. DepSign · p. 2

A. Conditions

  • C1 returns a severity label, while C2 first reasons about symptoms and then supplies the label.C3 marks each PHQ-9 criterion present, absent, or unclear, quotes the post for present criteria, and assigns the label by counting; C4 repeats this approach with BDI-II’s 21 items.Source: A. Conditions · pp. 2–3
  • C2S reasons over PHQ-9 criteria in C2’s format, letting the model assign the label; its criterion statuses can also be counted using C3’s rule.This separates reasoning over criteria from changing who assigns the final label.Source: A. Conditions · p. 3

B. From criteria to a label

The paper maps counts of present criteria or items to ordinal labels using either fitted or label-free thresholds. Chain-of-thought improves on direct prompting in most model–corpus pairs, but the results do not show a consistent criteria-extraction advantage under a priori thresholds.

B. From criteria to a label

  • Fitted thresholds maximize κw on a 600-post training split, while a priori thresholds use no labels and set PHQ-9 cutoffs from DSM-5 criteria.For nine PHQ-9 criteria, the DSM-5 rule sets the top threshold at 4.5; intermediate thresholds divide the remaining counts for DepSeverity’s four labels.Source: B. From criteria to a label · p. 3

5 thresholds

  • On DepSeverity, models mark no criterion in most posts; on DepSign, counts spread across the scale.Figure 1 shows criteria marked present per test post by model alongside a priori thresholds and assigned labels.Source: §5 · p. 3, Figure 1 · p. 3
  • Requiring seven symptoms for a SEVERE label catches no gold-SEVERE posts on either corpus and does not significantly beat C2 for frontier models on DepSign.On DepSeverity, κw falls to 0.231 and 0.223 for the frontier models and 0.117 for the 9B model.Source: §5 · p. 3

B. From criteria to a label

  • Fitted-threshold comparisons with zero-shot C1 or C2 are confounded because extraction alone uses labeled examples to set its decision thresholds.The study uses extended thinking off and reports bootstrap intervals paired across conditions on the same test posts.Source: §5 · p. 3, C. Models and protocol · p. 3

A. Positive control

  • C2 significantly beats C1 in five of six model–corpus pairs; the exception is the 9B model on DepSign, where it scores −0.050.Direct prompting over-predicts severity, and chain-of-thought reduces that share for the frontier models but not consistently for the 9B model.Source: A. Positive control · p. 3

B. The unfair comparison

Fitted thresholds can make criteria extraction look stronger than chain-of-thought, but the advantage largely disappears when the comparison uses a priori thresholds or gives both methods the same labels. The remaining 9B-model effects vary by corpus and questionnaire, with calibration accounting for part—but not all—of the observed gains.

B. The unfair comparison

  • Across nine a priori comparisons, criteria extraction shows no significant κw advantage over C2; three losses lose significance after accounting for run-to-run variation.Two losses are significant before that adjustment, and a third is barely so.Source: B. The unfair comparison · p. 4, Figure 2 · p. 4
  • With fitted thresholds, C3 scores above C1 and C2 in four of six pairs, significantly beating C2 for Qwen3.5-9B on both corpora (+0.149 and +0.152).The fitted thresholds add +0.167, +0.103 and +0.104 κw over a priori thresholds for the 9B, DeepSeek and Claude models on DepSeverity, respectively.Source: B. The unfair comparison · p. 4
  • For Qwen3.5-9B on DepSeverity, fitted thresholds [0.5, 1, 1.5] make any detected symptom sufficient for a higher-than-MINIMUM label, despite a mean count of 0.26 of nine symptoms.The authors conclude that the thresholds do the work.Source: B. The unfair comparison · p. 4

C. Giving C1 and C2 the same supervision

  • For frontier models, recalibrated C2 barely changes in three of four cases, and fitted C3 does not significantly beat it in any case.The authors describe C2 as already largely calibrated, while C3 needs labels.Source: C. Giving C1 and C2 the same supervision · p. 4
  • For Qwen3.5-9B on DepSeverity, equal-label recalibration removes a quarter of C3’s +0.149 gap; +0.109 remains after capping C3’s count at four levels.With a priori thresholds, PHQ-9 extraction reaches 0.322, below recalibrated C2, while BDI-II extraction reaches 0.470, a +0.091 difference that falls short of significance.Source: C. Giving C1 and C2 the same supervision · pp. 4–5

C. Giving C1 and C2 the Same Supervision

  • For DeepSeek-V4.1-Flash, recalibration raises C1 from 0.219 to 0.330 on DepSeverity and from 0.109 to 0.246 on DepSign, while DepSeverity C2 at 0.504 beats labeled C1.The paper concludes that C1’s weakness is calibration and that extraction adds nothing measurable over chain-of-thought in this comparison.Source: C. Giving C1 and C2 the same supervision · p. 5
  • On DepSign, recalibrated chain-of-thought narrows Qwen3.5-9B’s fitted-C3 lead to a non-significant +0.045, while recalibrated direct prediction still trails C3 by +0.078.The C2 map never predicts SEVERE; the authors conclude that labels close the gap through chain-of-thought but not direct prediction.Source: C. Giving C1 and C2 the same supervision · p. 5

D. Other explanations

Alternative explanations do not remove the frontier models’ lack of an a priori extraction advantage: more extracted symptoms, prompt changes, and a more flexible aggregation rule do not yield a consistent win. The C2S ablation points instead to the labeler, while criteria counting can lose severe cases, especially for the 9B model.

D. Other explanations

  • On DepSign, frontier models extract means of 2.31 and 2.29 symptoms versus 0.45 and 0.46 on DepSeverity, yet still show no advantage; their quoted evidence is 97.7–100% verbatim.The 9B model’s quotes match 92.4% and 80.9% of the time, or 90.4% and 81.9% in C4.Source: D. Other explanations · p. 5
  • Excluding 53 removed DepSign posts lowers every score, but the 9B model’s a priori gain remains significant at +0.115.The frontier fitted-C3 gap widens to about −0.051 for both models and remains non-significant.Source: D. Other explanations · p. 5
  • With BDI-II’s 21 items, C4 beats C2 only for Qwen3.5-9B: by +0.130 on DepSeverity and +0.108 on DepSign under the DSM-5 rule.Frontier-model C4 reaches C2 but does not beat it on either corpus; BDI-II’s own rescaled bands perform worse for frontier models and 9B on DepSeverity.Source: D. Other explanations · p. 5
  • For DeepSeek on DepSeverity, a permissive prompt finds 60% more symptoms but lowers fitted κw from 0.526 to 0.507; neither alternative prompt beats chain-of-thought.The symmetric prompt reaches 0.555 but remains below C2 with a priori thresholds by −0.071, not significantly.Source: D. Other explanations · p. 5
  • A more flexible ordinal model remains below the fitted-threshold count, while inverse-frequency weighting improves κw to 0.421–0.454 but misses fewer severe posts.The weighted model misses 14–26 of 56 SEVERE posts, compared with 43–49 for the count.Source: D. Other explanations · p. 5

E. Separating structure, format and labeler (C2S)

  • For frontier models, reasoning over criteria changes κw by −0.019 to +0.025 without significance, and changing output format likewise produces no significant difference.On the same C2S status lines, the a priori count is never significantly better than the model’s label and is once significantly worse (−0.084).Source: E. Separating structure, format and… · p. 5
  • For Qwen3.5-9B, reasoning reduces SEVERE predictions, while counting its C2S criteria misses 55–56 of 56 DepSeverity and 37–44 of 50 DepSign severe posts.The format effect varies by corpus: free-text status lines beat C3 on DepSeverity by +0.103 a priori but trail it on DepSign.Source: E. Separating structure, format and… · p. 5, Figure 3 · p. 6

F. Ordinal agreement versus missed severe posts

Higher ordinal agreement often coincides with more missed severe cases, and repeated runs add uncertainty to some comparisons. Human relabeling and feature-based analyses distinguish the two corpora: DepSign labels are recoverable by a reader, while DepSeverity labels are harder to explain through clinical criteria alone.

F. Ordinal agreement versus missed severe posts

  • Across twelve C1-to-C2 and C2-to-C3 comparisons, missed-SEVERE shares rise in eleven, significantly in eight before correction; higher κw often means more severe cases missed.For DeepSeek on DepSign, C3 nearly doubles κw from 0.109 to 0.203 but misses 49 of 50 SEVERE posts, compared with 9 under C1.Source: F. Ordinal agreement versus missed… · p. 6
  • For Qwen3.5-9B, C4 misses 29 of 50 DepSign severe posts and 49 of 56 on DepSeverity; on DepSign, frontier C4 misses fewer than C3 but more than chain-of-thought.Macro-F1 mostly agrees with κw, but the DepSign frontier C3 gain is confined to the majority MODERATE class and worsens both smaller classes.Source: F. Ordinal agreement versus missed… · p. 6

G. Run-to-run stability

  • Accounting for hosted-model run-to-run variation removes significance from three DSM-5-rule losses, while the 9B gains and all six C2−C1 effects remain significant.Across identical runs, C2 changes 105 and 132 of 706 labels, while C3 changes 21 and 29.Source: G. Run-to-run stability · p. 6

H. Can a human reader recover the labels?

  • On DepSign, the annotator reaches κw 0.416 [0.248, 0.567], but the paired difference from C2 is too uncertain to establish equivalence.The annotator’s interval is above zero, which the authors interpret as evidence that a reader following clinical criteria can recover DepSign labels.Source: H. Can a human reader recover the… · p. 6
  • On DepSeverity, the annotator scores 0.199 versus the best model’s 0.485 and reads 13 of 25 gold-SEVERE posts as showing no depressive content.With one annotator and no original guidelines, the authors cannot distinguish label noise from mismatch between their rubric and the original annotators’ rubric; collapsing labels reaches at most 0.244.Source: H. Can a human reader recover the… · pp. 6–7

I. What predicts the labels

  • For DepSeverity, Dreaddit’s non-text fields reach κw 0.333 and adding text-derived features reaches 0.404; neither frontier C3 with a priori thresholds differs significantly from the full-feature model.The authors call Dreaddit’s features a high lower bound and warn that this benchmark can reward systems without modeling depression severity.Source: §I · p. 7

VI. LIMITATIONS

The conclusion finds no significant frontier-model advantage for criteria extraction, while the limitations bound the results by corpus composition, exploratory analyses, and untested variants. The authors caution that ordinal agreement alone can obscure missed severe cases and does not establish clinical suitability.

VI. LIMITATIONS

  • DepSeverity has no depression community, while DepSign has only 706 test posts and 50 severe cases, limiting scope and statistical power.The eighteen main comparisons are corrected together with Holm’s method; other analyses, including C4 and missed-severe counts, are exploratory and uncorrected.Source: §VI · p. 7
  • Binary criterion scoring discards symptom intensity, and the study did not test graded extraction or larger local models.The authors note that graded extraction could narrow the gap to chain-of-thought, while the single small open-weights model leaves the calibration exception’s generality uncertain.Source: §VI · p. 7

VII. CODE, DATA AND ETHICS

  • The repository supports recomputation of nearly all reported numbers but omits post text because neither corpus has a stated license.Seven structured calls omitted one item; counting it as absent changed no reported κw by more than 0.001.Source: §VII · p. 7
  • The authors state that the systems are not fit for clinical use because the best ordinal-agreement systems miss the most severe cases.They recommend reporting error direction, such as per-class recall or expected cost under asymmetric error costs.Source: §VII · p. 7

VIII. CONCLUSION

  • Frontier criteria extraction shows no significant κw advantage over chain-of-thought across two corpora, two questionnaires, and both threshold regimes.The conclusion also says its apparent gains depend on fitted thresholds and that its macro-F1 gains on DepSign come from the majority class.Source: §VIII · p. 7
  • For the 9B model, a priori structure helps only on DepSign with either questionnaire and on DepSeverity with BDI-II, while gains come with more missed severe cases.Holding criteria fixed indicates that counting, rather than reasoning over criteria, loses severe cases; extraction outputs are more stable across identical runs and auditable criterion by criterion.Source: §VIII · p. 8
  • The paper advises reporting both a priori and fitted thresholds, direct prediction and chain-of-thought, and the direction of errors alongside agreement.For the 9B model on DepSign, comparison only with chain-of-thought gives +0.143, whereas direct prediction gives +0.093.Source: §VIII · p. 8

APPENDIX A ALL COMPARISONS

The appendices document how paired comparisons are classified, how criterion-count thresholds are set, and how predictions are recalibrated to test equal supervision.

APPENDIX A ALL COMPARISONS

  • Table III reports paired DepSeverity Δκw intervals; C marks main tests corrected together, H marks Holm survivors, and bold intervals exclude zero before correction.C4 uses the DSM-5 rule for its a priori thresholds.Source: Table III · p. 9, Appendix A · p. 9

APPENDIX B THRESHOLDS

  • On DepSeverity, fitted C3 top thresholds fall to 1.5–2.5 versus five symptoms under the DSM-5 rule; fitted C4 top thresholds on DepSign rise to 6.5–10.5 items.The passage says the lower DepSeverity thresholds mean two or three marked criteria can yield a severe label, while the higher DepSign C4 thresholds catch fewer severe posts than the DSM-5 rule.Source: Appendix B · p. 9

APPENDIX C GIVING C1 AND C2 THE SAME SUPERVISION

  • Tables VI–VII test monotone relabeling fitted on the 600-post split, comparing C3 with recalibrated predictions and extending the test to a priori rules and C4.Table VI lists changed predicted labels and reports exploratory C3-minus-recalibrated-condition Δκw intervals.Source: Appendix C · p. 9, Table VI · p. 11, Table VII

APPENDIX D PER-CLASS RECALL

Per-class recall reveals that higher ordinal agreement can trade severe-class recall for majority-class recall, while a learned aggregator improves severe-case recovery relative to fitted counts.

APPENDIX D PER-CLASS RECALL

  • On DepSign, fitted-threshold C3 raises MODERATE recall to 0.68–0.88 but cuts SEVERE recall to 0.02–0.28; on DepSeverity, the weighted learned aggregator misses 14–26 of 56 severe posts versus 43–49 for fitted counts.Tables VIII–IX give recall for every class; Table VIII covers DepSeverity test posts with class counts 513, 58, 79, and 56.Source: Appendix D · p. 12, Table VIII · p. 12

APPENDIX E OUTPUT LENGTH

Output length differs across prompting conditions, but the condition ranking does not follow length.

APPENDIX E OUTPUT LENGTH

  • C2 writes two to three times as many tokens as C3, C4 writes about as much as C2, and C2S is shorter than C2 for every model.C2 discusses absent as well as present symptoms; C2S walks through C3’s criteria, yet the ranking of conditions does not track output length.Source: Appendix E · p. 12

APPENDIX F PROMPT VARIANTS

Appendix F supplies the full C3 prompt-variant results and tables reporting per-class recall and test-call token counts.

APPENDIX F PROMPT VARIANTS

  • Table IX reports per-class recall on the DepSign test set, with class counts of 184, 472, and 50.Bold indicates the highest recall per class within each model.Source: Table IX · p. 13
  • Table X reports mean input and output tokens per test call, as reported by each provider.Source: Table X · p. 13
  • Table XI compares C3 prompt variants with C2 on DeepSeek-V4.1-Flash using Δκw and 95% paired-bootstrap intervals.Bold intervals indicate comparisons whose intervals exclude zero; the caption does not provide the cell values.Source: Appendix F · p. 12, Table XI · p. 13

APPENDIX G PROMPTS

Appendix G gives the prompts used to rate depression severity directly, reason through symptoms, or extract symptom criteria. The extraction prompts specify symptom-status labels and, in some variants, supporting text spans.

APPENDIX G PROMPTS

  • Direct prompting requests only one severity label, while chain-of-thought prompts first ask the model to reason about symptoms and then state severity.Source: Appendix G · p. 14
  • PHQ-9 extraction requests present, absent, or unclear for nine symptoms and a verbatim evidence span when a symptom is present.The prompt says silence about a symptom should be labeled unclear, not absent.Source: Appendix G · pp. 14–15
  • BDI-II extraction uses the same three status labels for 21 symptoms and requests a verbatim evidence span for present symptoms.Source: Appendix G · p. 15
  • The permissive variant counts direct statements, paraphrases, or reasonably implied symptoms as present and errs toward present when supported.Source: Appendix G · pp. 15–16
  • The symmetric variant requires supporting text spans for both present and absent judgments; without a supporting span, the prompt specifies unclear.Source: Appendix G · p. 16
Loading 2609.39049v2…