PaperlayerView exact revision on arXiv

Source-linked AI summary

The Divergence Hypothesis: Unmasking Lexical Interference and Label Bias in Mental Health NLP

Moustafa Yehia Hassan

arXiv:2608.20353v1cs.CLcs.AI

TL;DR

CMH classifiers may learn different signals from human and distant-supervision labels, complicating distribution-shift claims. The paper introduces TSS to decompose those signals and DoD to quantify divergence, finding lexical interference on human-labeled data but not auto-labeled data. Masking results further indicate that style performance can largely survive content-word deletion, positioning TSS as an audit framework rather than a clinical screening tool.

  • Problem

    CMH benchmarks can mismatch their labels with the construct they are taken to measure, allowing different label sources to reward lexical shortcuts or stylistic signals.

  • Method

    TSS decomposes text into lexical, mostly content-free morpho-syntactic, and 154-feature psycholinguistic style channels, while DoD quantifies label-source divergence with bootstrap inference.

  • Results

    Adding lexical features to style reduces Macro-F1 on human-labeled data by 0.072 (p < 10^-4) but not on auto-labeled data, with DoD(BC-A) = 0.0374, 95% CI [0.0097, 0.0651], p = 0.0032.

  • Takeaways & Limitations

    TSS flags label-source-specific shortcut learning before generalization claims are made and shows that style performance can largely survive deliberate content-word deletion.

  • Takeaways & Limitations

    The four-dataset design does not identify causal label-source effects; a controlled within-platform dual-annotation study remains necessary.

Abstract

from arXiv · show

Computational mental health (CMH) classifiers often degrade under distribution shift because human annotators and distant-supervision pipelines reward different linguistic signals. We introduce TSS (Triple-Stream Stress probe), a multi-channel diagnostic framework that decomposes text into (A) lexical character n-grams, (B) a small, mostly content-free morpho-syntactic channel, and (C) a 154-feature psycholinguistic style channel. Across four English datasets (N=12,906), TSS reveals a lexical interference effect: adding lexical features to the style channel reduces Macro-F1 on human-labeled data (mean drop 0.072, p<10^-4) but not on auto-labeled data. We propose Degree of Divergence (DoD), a difference-in-differences statistic adapted from econometrics for label-source auditing, with instance-level bootstrap inference; the headline estimate is DoD(BC-A) = 0.0374, 95% CI [0.0097, 0.0651], p=0.0032. A platform-stratified Twitter-only DoD (which removes the Reddit vs. Twitter contrast) reproduces the pattern with bootstrap inference: DoD-Tw(BC-A) = +0.096 (p<0.001) and DoD-Tw(AC-A) = -0.089 (p<0.001). Interventional masking (pos_only) retains ~95-99% of Channel C's performance after destroying content words on human datasets, indicating that the style channel does not rely primarily on lexical surface form. TSS is positioned as a diagnostic audit framework, not a clinical screening tool: it flags label-source-specific shortcut learning before generalization claims are made.

1 Introduction

CMH benchmarks can conflate label-source signals, allowing lexical shortcuts to substitute for robust psycholinguistic markers. This paper frames stress detection as an audit of which linguistic channels human and distant-supervision labels reward.

  • Distant-supervision proxies can systematically reward lexical cues, while human annotators often rely on stylistic and morpho-syntactic signals.
  • Conflating label sources can produce shortcut learning, where models exploit spurious lexical correlations instead of robust psycholinguistic markers.
  • The task is binary stress detection from short social-media posts, with stress defined as a dataset-level distress label rather than a clinical diagnosis.
  • The study asks whether label source changes rewarded linguistic channels, whether lexical content interferes with stylistic features, and whether the gap can be quantified.
  • TSS decomposes text into lexical, mostly content-free morpho-syntactic, and psycholinguistic style channels for length-robust auditing.

2 Related Work

Prior CMH research links distant-supervision labels and distribution shift to lexical shortcut learning, while psycholinguistic work motivates stylistic signals as indicators of cognitive style and distress.

  • Function words and stylistic patterns have been studied as reflections of cognitive style more reliably than consciously controlled topical content.
  • Psycholinguistic frameworks identify absolutist language and cognitive distortions as linguistically measurable correlates of distress.
  • Distant-supervision pipelines based on diagnoses or community membership can encourage surface lexical cues and degrade under distribution shift.
  • The experimental backbone uses Dreaddit as a structured human-labeled benchmark and Twitter mental-health corpora as distant-supervision proxies rather than stress gold standards.

3 Data and Task

The study evaluates binary stress labels across four English datasets spanning human and distant-supervision sources and Reddit and Twitter platforms, with controls for leakage and imbalance.

  • The evaluation covers four English datasets after within-dataset exact-match deduplication, with Table 1 summarizing platform, size, label source, and intended use.
  • Twitter-auto uses diagnosis self-reports as an auto-positive proxy, while Twitter-gold manually targets perceived psychological distress rather than mental-health keywords.
  • Positive prevalence is 0.516 for Dreaddit-test, 0.338 for Twitter-gold, 0.478 for Twitter-auto, and 0.880 for Reddit-combi.
  • Reddit-combi is an internally assembled Reddit corpus under distant supervision and is used exclusively as an auto-label proxy.
  • Exact-match cross-dataset decontamination removes overlapping training posts, while Macro-F1 and PR-AUC address severe class imbalance.
  • Paired comparisons use McNemar’s test and paired bootstrap confidence intervals with Benjamini–Hochberg false-discovery control.
  • Dreaddit’s five stressor domains enable leave-one-domain-out evaluation of cross-stressor generalization beyond platform transfer.

4 The TSS Probe

TSS is an interpretable diagnostic decomposition that contrasts lexical, morpho-syntactic, and psycholinguistic channels rather than optimizing competitive classification. Its design includes content masking, normalization, and DoD auditing to test shortcut behavior across label sources.

  • The TSS Probe: TSS compares three channels—lexical A, mostly content-free morpho-syntactic B, and psycholinguistic style C—to isolate label-source-specific signal preferences.
  • Channel A: Channel A uses character-level TF–IDF n-grams of length 3–5 with χ2 feature selection at k = 500.
  • Channel B: Channel B converts POS bigrams and abstract POS-SVO triples into six features, five of which discard lexical tokens.
  • Channel B: Channel B serves as a near-content-free diagnostic and structural stress test, not as a performance booster combined with C.
  • Channel C: Channel C contains 154 stylistic features, including length-robust transforms, Yule’s I lexical diversity, fragmentation, punctuation bursts, and negation scope.
  • Degree of Divergence: DoD is a difference-in-differences statistic estimated with instance-level bootstrap inference to quantify label-source divergence, including a Twitter-only platform-stratified variant.
  • Interventional masking: The masking suite intervenes on text with pos_only, content_only, function_only, random_pos, and an unmasked condition to test which signals survive lexical destruction.

5 Results

Across human- and auto-labeled datasets, TSS exposes label-source-dependent channel behavior: lexical augmentation harms human-label performance, while style and structural channels remain comparatively stable. Platform-stratified DoD and masking analyses support label-source auditing while retaining important scope caveats.

  • 5.1 Channel Performance: 0.690 Macro-F1 for BC on both Dreaddit-test and Twitter-gold indicates nearly identical human-labeled performance across platforms.The reported platform difference is ∆≈0.0002.
  • 5.1 Channel Performance: C falls from 0.701 to 0.504 and BC from 0.690 to 0.455 when moving from human-labeled Twitter-gold to auto-labeled Twitter.The passage interprets this pattern as consistent with auto labels rewarding lexical proxy cues that structural and style probes less readily mimic.
  • 5.2 Lexical Interference Effect: 0.072 Macro-F1 is the mean human-labeled drop from augmenting C with A, versus −0.020 on auto-labeled data.The human-labeled effect has 95% CI [0.052, 0.092] and one-sided p < 10^-4; the auto-labeled test for drop > 0 has p=0.9988.
  • 5.3 Degree of Divergence: 0.0374 is the BC–A DoD estimate, with 95% CI [0.010, 0.065] and p=0.0032, while AC–A is −0.0453 with p<0.001.The full DoD table presents these as contrasting channel effects across label sources.
  • 5.4 Platform-Stratified Twitter-Only DoD: Twitter-only DoD preserves the cross-platform pattern: structural channels gain relative to A under human labels and lose under auto labels.The paired bootstrap uses Nboot=2,000; the result makes a pure platform-shift explanation unlikely, though annotation and sampling differences remain.
  • 5.5 Interventional Masking: On Dreaddit-test, C/pos_only is indistinguishable from C/none (∆=−0.006, p=0.72), while BC/pos_only exceeds BC/none (∆=+0.037, p=0.013).Random positional scrambling causes significant Twitter-gold drops for C and BC, providing a destructive control for the masking intervention.
  • 5.6 Baselines and Robustness: TSS is a diagnostic audit rather than a claim of competitive superiority or clinical screening validity.LLaMA-3-8B reaches Macro-F1 0.800 on Twitter-gold, and the paper frames baselines as epistemic diagnostics rather than direct competitors.

6 Discussion

TSS frames strong lexical performance as potentially shortcut-driven when topical stress vocabulary overlaps with structural distress markers. The discussion argues that relative channel differences across label sources, rather than absolute performance, reveal this divergence and that structural features can recover failures caused by lexical shortcuts.

  • TSS is a diagnostic tool that uses DoD to summarize label-source-specific performance gaps before generalization claims.Its purpose is auditing shortcut learning, not serving as a competitive classifier.
  • 6.1 Why Lexical Baselines Look Strong: In auto-labeled or closed-community settings, topical stress vocabulary can overlap with structural distress markers, allowing models to detect correlated topics rather than distress itself.This semantic–structural intersection helps explain strong baseline performance on auto-labeled data.
  • 6.1 Why Lexical Baselines Look Strong: +0.008 Macro-F1; p = 0.259: adding morpho-syntax to C on Reddit-combi yielded no detectable gain, consistent with channel overlap under high SSI.C still exceeded A by +0.070 (p < 0.0001), so relative channel advantages remain the more informative diagnostic.
  • 6.4 Baseline Interpretation: On Twitter-gold, TSS-C approached MentalBERT but did not establish overall competitive superiority because LLaMA-3-8B reached Macro-F1 0.800.The authors interpret this as representation efficiency relative to one domain-pretrained transformer baseline, not a SOTA claim.
  • 6.2 Error Analysis: Across 800 audited cases, Channel C corrected 253 Channel-A errors and BC corrected 241, covering both keyword-driven false positives and implicit-stress false negatives.The two error types show lexical shortcuts failing in opposite directions.
  • 6.3 Robustness: Channel C varied little across held-out stressor domains, with standard deviation 0.015, supporting its robustness across heterogeneous stressor causes.The paper treats social-media data as a stress-test environment rather than a clinical stand-in.

7 Conclusion

The conclusion presents TSS as an audit framework for testing whether label sources reward different linguistic evidence. Across four English datasets, lexical features interfered with style-based performance on human-labeled data, while DoD and masking supported a label-source-specific interpretation.

  • TSS tests whether different label sources reward different linguistic evidence in computational mental health classification.
  • Adding lexical surface features to style features reduced human-labeled Macro-F1 by 0.072 on average (p < 10^-4) but not on auto-labeled datasets.The headline DoD estimate was DoD_BC–A = 0.0374 (p = 0.0032).
  • Twitter-only DoD reproduced the pattern, and Dreaddit masking found Channel C statistically unaffected by full content-word deletion (∆ = −0.006, 95% CI [−0.037, +0.023], p = 0.72).A controlled within-platform dual-annotation study remains necessary for causal identification.
  • TSS is an audit workflow that flags label-source-specific shortcut learning before generalization claims are made.

Limitations

The paper’s evidence is constrained by its datasets, annotation protocols, feature-level analysis, lexical baseline choices, and nonclinical scope. These boundaries limit how broadly the findings and interpretations should be applied.

  • Evaluation covers four English datasets and a binary stress label, while Twitter-gold lacks inter-annotator agreement under the available manual protocol.Twitter-auto also inherits disorder→stress conflation from its distant-supervision source.
  • The label-source and platform factors are not fully separated; Twitter-only DoD removes the most obvious confound, but causal identification requires within-platform dual annotation.
  • No feature-level within-channel ablation was performed, limiting interpretability of individual features inside Channel C.The study instead evaluates the channel-level diagnostic decomposition.
  • A word-unigram/bigram lexical baseline was omitted, so future comparisons must test whether the lexical interference effect is specific to character-level tokenization.
  • TSS is not a clinical diagnostic tool, and clinical deployment would require clinician oversight, informed consent, and risk management.DoD may also require calibration for morphologically richer languages such as Arabic and Chinese.
  • LLaMA-3-8B is reported only in a 3-shot diagnostic setting, so its instability is not evidence about fine-tuned LLM performance.

Ethical Considerations

The study is not a diagnostic medical tool and focuses on auditing label bias and methodological rigor in computational mental-health NLP.

  • The study is not a diagnostic medical tool.It audits label bias and methodological rigor in CMH NLP rather than providing clinical screening.

Appendices

The appendices provide mathematical formulations, statistical tables, robustness audits, qualitative analyses, extended metrics, and SHAP audits supporting the main text.

  • The appendices provide mathematical formulations, full statistical tables, robustness audits, qualitative case analysis, extended metrics, and SHAP audits.
  • A.1 Degree of Divergence: DoD compares Macro-F1 under human and auto label sources using instance-level bootstrap inference with n=10,000.DoD is adapted from causal econometrics.
  • Yule’s I is capped at 10^3 to prevent extreme outliers from very short texts.It is preferred over TTR because TTR varies mathematically with text length, which can bias cross-platform comparisons.
  • The appendices report paired baseline comparisons, channel contrasts, leave-one-domain-out robustness, per-stressor breakdowns, and paired effect sizes.

D Qualitative Conflict Zone

The qualitative conflict-zone analysis examines disagreements between lexical and structural channels, quantifies correction patterns, and documents representative failure modes and robustness diagnostics.

  • D Qualitative Conflict Zone: 800 instances, 200 per dataset, were extracted for qualitative analysis of disagreements between lexical and structural channels.The full workbook was released with the code.
  • D Qualitative Conflict Zone: 682 cases show Channel B correcting Channel A errors: 60.9% are keyword-driven false positives and 39.1% are implicit-stress false negatives.
  • D Qualitative Conflict Zone: Channel C alone corrects 253 of Channel A’s errors, while BC corrects 241.The Channel C corrections include 195 keyword-driven false positives and 58 implicit-stress cases.
  • D Qualitative Conflict Zone: Table 21 presents eight keyword-driven false positives and five implicit-stress cases where lexical and structural verdicts diverge.
  • D Qualitative Conflict Zone: Channel A’s top 10 features, only 2% of its feature space, account for 33.3% of absolute SHAP weight.This is 16 times the uniform-contribution expectation.
Loading 2608.20353v1…