PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media

Rajveer Singh Pall, Sameer Yadav

arXiv:2608.26138v1cs.CLcs.LG

TL;DR

Mental health NLP models may not remain reliable across social-media platforms with different users, norms, vocabulary, and expressions of distress. This paper introduces CPFE, a five-axis audit of transformer models across platforms, finding co-occurring failures in discrimination, calibration, attribution stability, and prediction equity that support cross-platform validation.

  • Problem

    Evidence is limited on whether mental health NLP models trained on one social-media distribution remain reliable across platforms with different users, norms, vocabulary, and distress expressions.

  • Method

    The study introduces CPFE, a five-axis audit, and applies it to four transformer classifiers trained on Kaggle mental-health data and evaluated on Reddit and Twitter proxy labels.

  • Results

    30.3–35.4% Reddit AUC degradation and 37.9–39.5% Twitter degradation co-occurred with calibration, attribution, and prediction-equity failures across independently evaluated models.

  • Takeaways & Limitations

    Cross-platform validation across all five CPFE axes should be treated as a standard requirement for mental health NLP systems intended for heterogeneous deployment environments.

  • Takeaways & Limitations

    Emotion-to-clinical label mappings are heuristic, so human-annotated Reddit and Twitter clinical labels are needed to disentangle construct mismatch from distributional failure.

Abstract

from arXiv · show

We introduce the Cross-Platform Fairness Evaluation (CPFE) framework -- a five-axis audit protocol covering discriminative performance, calibration, statistical significance, prediction equity, and attribution stability -- and apply it to four transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on a Kaggle mental health corpus (n=35,556) and evaluated on Reddit (n=6,257) and Twitter (n=2,883) test sets with emotion labels mapped to clinical proxies. All three independently evaluated models exhibit consistent and substantial cross-platform AUC degradation (30.3-35.4% on Reddit, 37.9-39.5% on Twitter) relative to within-platform performance (AUC 0.983-0.987), confirmed across five independent training seeds. Calibration failure is concurrent and severe: ECE rises from 0.056-0.060 in-domain to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter. Platform-specific temperature scaling reduces mean ECE by 88.0% without altering discriminative performance (mean |delta AUC|<0.01), confirming separable failure modes. Prediction equity analysis reveals large cross-platform disparities (raw DI < 0.17; prior-shift-adjusted DI: 0.11-0.29 on Reddit), with equalized odds differences of 0.753-0.830 for mental health proxy classes on Reddit and 0.755-0.831 for anxiety on Twitter. Attribution stability analysis shows near-complete vocabulary divergence across platforms (Jaccard J=0 in 14/16 model-class pairs at K=10). These findings support treating cross-platform validation across all five CPFE axes as a standard requirement for mental health NLP systems in heterogeneous environments. In a single-seed fine-tuning experiment, mean AUC improved by 0.216, suggesting target-platform labels provide greater benefit as training signal than as calibration signal.

1 INTRODUCTION

Mental health NLP models face high-stakes reliability risks when social-media deployment distributions differ from training data. This study introduces CPFE, a five-axis framework for auditing cross-platform performance, calibration, significance, equity, and attribution stability.

  • Motivation: Mental health NLP is high-stakes because false negatives may delay review of distress language, while false positives can create alert fatigue.Findings concern proxy emotion labels mapped to clinical categories, not operational clinical diagnoses.
  • Problem: Platform differences in demographics, communication norms, vocabulary, text length, and distress expression can make models learn platform-specific rather than generalisable signals.Single-platform evaluation can therefore produce optimistic estimates that fail under deployment shifts.
  • Framework: CPFE audits five axes: discriminative performance, probabilistic calibration, statistical significance, prediction equity, and attribution stability.The axes use AUC/F1, ECE, bootstrap-corrected pairwise AUC tests, Symmetric Disparate Impact/Equalized Odds Difference, and top-K vocabulary Jaccard similarity.
  • Study design: The study applies CPFE to BERT, RoBERTa, Emotion-DistilRoBERTa, and GoEmotions-RoBERTa across Kaggle, Reddit, and Twitter datasets mapped to four clinical categories.The models span general-domain, emotion-fine-tuned, and GoEmotions-pretrained regimes.
  • Contributions: 30.3–39.5% cross-platform AUC degradation and J=0 in 14/16 model-class pairs at K=10 demonstrate substantial performance and attribution instability.Platform-specific temperature scaling reduces mean ECE by 88% while leaving discriminative performance unchanged.

2 RELATED WORK

Prior work establishes distribution shift, external validation, calibration, and fairness auditing as important concerns for NLP and clinical prediction. This study extends cross-platform mental health classification research by evaluating additional fairness axes and comparing remediation strategies.

  • Distribution shift: Distribution shift has been studied in sentiment analysis, cross-platform mental health classification, and clinical text mining, but mitigation methods are rarely evaluated in mental health settings.Proposed mitigations include domain-adaptive pretraining, adversarial domain adaptation, and self-training on unlabelled target-domain text.
  • Mental health NLP: Transformer classifiers show strong within-platform discrimination for social-media depression, anxiety, and stress proxies, although proxy-label clinical validity remains contested.Critical reviews identify limited external validation, clinical label ambiguity, and selection bias as methodological limitations.
  • Cross-platform evaluation: Kula et al. provide the most directly relevant cross-platform study across Reddit and Twitter; this work adds calibration, corrected significance testing, platform-stratified equity, attribution stability, and remediation comparisons.The remediation comparison contrasts temperature scaling with target-domain fine-tuning.
  • Validation and calibration: Clinical prediction guidance requires external validation and calibration before deployment, with ECE and reliability diagrams standard under TRIPOD reporting guidelines.Prior work also finds modern deep neural networks systematically overconfident and identifies temperature scaling as an effective post-hoc remedy.
  • Platform-as-group fairness: Disparate impact and equalized odds are foundational fairness criteria, which this study extends to source platforms rather than demographic groups in multiplatform mental health deployment.Prior NLP fairness audits have primarily focused on demographic attributes.

3 DATA AND PREPROCESSING

The study combines three emotion-labelled social-media sources, remaps emotions to four broad mental-health categories, and evaluates platform-specific test partitions. The data are imbalanced and differ in label distributions and text length, creating confounds beyond domain shift.

  • Kaggle mental health corpus: n=35,556 samples formed the stratified training set, while the within-platform test set comprised n=7,620 samples across normal, depression, anxiety, and stress labels.The Kaggle partition was heavily skewed toward depression (56.6%), compared with anxiety (7.5%) and stress (7.2%).
  • Reddit evaluation data: n=6,257 held-out Reddit comments were mapped from 27 emotions plus neutral into depression, anxiety, stress, and normal clinical proxies.Four alternative mapping variants assessed robustness because the mappings approximate broad syndromes from surface emotion expressions.
  • Twitter evaluation data: n=2,883 held-out tweets were remapped into the same four categories, with Twitter distributed as 43.6% normal, 30.1% depression, 12.3% anxiety, and 14.0% stress.The source dataset contained 20,000 English tweets labelled with six emotions.
  • Cross-platform composition: 112,211 samples span all three sources, whose shifting label distributions constitute an independent confound in addition to domain shift.The combined corpus includes the Kaggle, Reddit, and Twitter sources.
  • Tokenization and truncation: 59.2% of Kaggle test-set samples exceed 64 tokens, compared with 0.0% of Reddit samples and 2.8% of Twitter samples.All texts used model-specific tokenizers with maximum sequence length 64; Kaggle texts averaged 118.3 words versus 13.5 for Reddit and 19.0 for Twitter.

4 METHODOLOGY

The methodology evaluates four transformer classifiers across five CPFE axes using standardized training, performance, calibration, fairness, attribution, and robustness procedures. Analyses include multi-seed replication, label-remapping sensitivity checks, post-hoc temperature scaling, and matched-data target-domain fine-tuning.

  • Model training: Four transformer models were fine-tuned as four-class sequence classifiers with AdamW, 2×10−5 learning rate, 0.01 weight decay, warmup, gradient clipping, and early stopping.Maximum sequence length was 64 tokens throughout.
  • Performance evaluation: Macro one-vs-rest AUC, macro F1, accuracy, per-class AUC confidence intervals, and bootstrap-based pairwise platform comparisons quantified discriminative performance and significance.Pairwise comparisons used B=2,000 bootstrap standard errors and Bonferroni-adjusted α′=0.0167 within each model’s three-comparison family.
  • Calibration: Calibration was measured with Expected Calibration Error using M=10 equal-width bins and bootstrap 95% confidence intervals from B=1,000 resamples.Post-hoc temperature scaling optimized negative log-likelihood on a stratified 10% target-platform calibration split.
  • Prediction equity: Prediction equity was assessed with symmetric DI and equalized odds difference, where DI<0.80 violates the four-fifths rule, DI<0.50 indicates severe disparity, and EOD=0 denotes perfect equalized odds.DI compares group-specific outcomes between Kaggle and the target platform, while EOD is based on true-positive rates.
  • Attribution stability: Attribution stability used Captum gradient-based token saliency, aggregated word-type scores from n=200 samples per platform and compared top-K sets with Jaccard similarity.Jaccard similarity was reported for K ∈{5, 10, 15, 20}.

5 RESULTS · 5.1 Within-Platform Performance · 5.2 Cross-Platform AUC Degradation (Axis 1)

Within-platform performance was strong and well calibrated, but cross-platform evaluation revealed substantial, systematic AUC and F1-macro degradation on Reddit and Twitter. Results were organized around the CPFE axes, with discriminative performance establishing the primary failure.

  • 5 RESULTS: Five CPFE axes were evaluated sequentially: discriminative performance established the primary failure, while statistical significance, prediction equity, calibration, and attribution stability addressed confirmation, consequences, correction, and mechanism.The CPFE framework is illustrated in Fig. 1.
  • 5.1 Within-Platform Performance: RoBERTa achieved the highest within-platform macro AUC (0.987) and macro F1 (0.883), while GoEmotions-RoBERTa reached AUC 0.985 and F1 0.871.Within-platform ECE was 0.056–0.060, indicating good calibration on the training distribution.
  • 5.2 Cross-Platform AUC Degradation (Axis 1): 30.3–35.4% AUC degradation occurred on Reddit and 37.9–39.5% on Twitter across the three independently evaluated models.Macro AUC fell from 0.983–0.987 within-platform to 0.637–0.685 on Reddit and 0.596–0.611 on Twitter.
  • 5.2 Cross-Platform AUC Degradation (Axis 1): 63–68% F1-macro reduction produced Reddit scores of 0.303–0.332, compared with in-domain performance of 0.862–0.883.This indicated that recall-precision balance across the four proxy classes was largely unachievable out of domain.
  • 5.2 Cross-Platform AUC Degradation (Axis 1): F1-macro of 0.303–0.332 on Reddit remained above the majority-class baseline of ≈0.224, but represented only marginal improvement.The majority-class classifier always predicted ‘normal’; its Twitter F1-macro was ≈0.266.
  • 5.2 Cross-Platform AUC Degradation (Axis 1): 28.6% was GoEmotions-RoBERTa’s smallest Reddit AUC drop, with AUC 0.703, but its result was non-independent because it was pretrained on GoEmotions.The Reddit result therefore represented an in-distribution ceiling rather than a cross-platform benchmark.
  • 5.2 Cross-Platform AUC Degradation (Axis 1): 37.9–39.5% Twitter AUC drops converged across all four models, with AUC 0.596–0.611, showing that GoEmotions pretraining did not transfer to Twitter.This contrasted with its Reddit advantage under the non-independent evaluation.

5.3 Per-Class AUC with DeLong Confidence Intervals · 5.4 Statistical Significance of Performance Gaps (Axis · 5.5 Disparate Impact and Equalized Odds (Axis 4)

Per-class performance was weakest for Twitter stress and Reddit depression proxies, while all pairwise AUC gaps were statistically significant. Reddit showed severe disparate impact and equalized-odds disparities that persisted after adjusting for platform prevalence differences.

  • 5.3 Per-Class AUC with DeLong Confidence Intervals: 0.522–0.542: Twitter stress AUC was only marginally above chance across all models.RoBERTa achieved AUC 0.522 [95% CI: 0.492, 0.551], whose lower bound fell below 0.500.
  • 5.3 Per-Class AUC with DeLong Confidence Intervals: 0.597–0.628: Reddit depression-proxy AUC fell below commonly cited clinical discrimination benchmarks.Reddit anxiety-proxy AUC was somewhat better preserved at 0.686–0.705.
  • 5.4 Statistical Significance of Performance Gaps (Axis: All 12 pairwise AUC comparisons were significant under Bonferroni-corrected bootstrap Z-tests.Kaggle–Twitter comparisons had Z-statistics of 47.7–51.9, while Reddit–Twitter comparisons had Z=2.89–8.25.
  • 5.5 Disparate Impact and Equalized Odds (Axis 4): 0.091–0.167: Reddit disparate-impact values for depression, anxiety, and stress were below both the 0.50 severe-disparity threshold and 0.80 four-fifths rule.Normal-class DI was 0.304–0.318, also below both thresholds.
  • 5.3 Per-Class AUC with DeLong Confidence Intervals: Five training seeds: Table 4 reports per-class one-vs-rest AUC with 95% DeLong confidence intervals across three platforms.The reported values are means across five training seeds; Reddit evaluation is noted as non-independent.
  • 5.5 Disparate Impact and Equalized Odds (Axis 4): 0.830: Reddit depression equalized-odds difference reached this value for BERT and RoBERTa.This corresponds to a depression-proxy true-positive rate 83 percentage points lower on Reddit than Kaggle.
  • 5.5 Disparate Impact and Equalized Odds (Axis 4): 0.11–0.29: Prior-shift-adjusted DI across Reddit mental-health proxy classes remained substantially disparate after reweighting to reference-platform prevalences.This indicates prevalence differences alone did not explain the observed disparity.
  • 5.5 Disparate Impact and Equalized Odds (Axis 4): EOD values remained severely elevated after conditioning on true class membership, making them less susceptible to prevalence-distribution confounding.The analysis distinguished genuine prediction-rate disparities from platform-level label-distribution differences.

5.6 Temperature Scaling Recalibration (Axis 2)

Temperature scaling sharply reduces cross-platform calibration error while leaving ranking-based discrimination unchanged. Thus, recalibration addresses overconfidence but not the underlying cross-platform AUC deficit.

  • Calibration improvement: T ∗=3.14–9.52: Optimal temperatures vary by model and platform, reflecting differing degrees of overconfidence.GoEmotions-RoBERTa on Reddit uses T ∗=3.14, while RoBERTa on Twitter uses T ∗=9.52.
  • Calibration improvement: 88.0%: Mean ECE falls from 0.364 before recalibration to 0.044 after, across all models and platforms.The reduction ranges from 66.3% to 96.4%.
  • Discriminative performance: Mean |∆AUC|=0.009: Macro AUC remains invariant to temperature scaling, with changes ranging from −0.010 to +0.033.The invariance follows because softmax(z/T) monotonically transforms scores without changing prediction rankings.
  • Discriminative performance: Post-hoc recalibration alone cannot address the cross-platform discriminative deficit.The calibration split used 10% of the target-platform test set, with ECE measured on the remaining 90%.

5.7 Attribution Stability (Axis 5) … 5.10 Truncation and Text Length

The audit finds near-complete cross-platform attribution divergence, while target-domain fine-tuning improves discrimination and cross-platform degradation remains under alternative label mappings. Large differences in source-text length and truncation further distinguish Kaggle from Reddit and Twitter inputs.

  • 5.7 Attribution Stability (Axis 5): J=0.000 occurs in 14 of 16 model-class pairs for Kaggle-to-Twitter top-10 attribution comparisons.RoBERTa-normal and Emotion-DistilRoBERTa-normal each show J=0.111.
  • 5.7 Attribution Stability (Axis 5): 0% of top-10 features match the 45-term proxy-clinical vocabulary on Reddit and Twitter, versus 0–30% on Kaggle.Kaggle attention concentrates on DSM-adjacent terminology, whereas Reddit and Twitter attribution shifts toward platform-specific conversational language.
  • 5.8 Fine-Tuning vs. Temperature Scaling: +0.216 mean AUC improvement was observed across model-platform pairs after single-seed target-platform fine-tuning.The Emotion-DistilRoBERTa Twitter result contributed ∆AUC = +0.350; excluding it, the mean gain was 0.197.
  • 5.9 Sensitivity Analysis: 26.8–34.7% Reddit drops and 35.3–39.0% Twitter drops persisted under binary Mapping B.Cross-platform AUC degradation was consistent across all four alternative label mapping schemes.
  • 5.9 Sensitivity Analysis: J=0 denotes complete top-10 attribution vocabulary divergence, while the random baseline is J≈0.0001 for a 50,265-token vocabulary.Table 8 covers 16 model-class pairs across four models and four classes.
  • 5.10 Truncation and Text Length: 118.3 mean words per sample occurred on Kaggle, compared with 13.5 on Reddit and 19.0 on Twitter.Kaggle’s median was 67.0 words and p75 was 154.0.
  • 5.10 Truncation and Text Length: 59.2% of Kaggle samples exceeded the 64-token maximum sequence length, versus 0.0% of Reddit and 2.8% of Twitter samples.The estimate covered n=4,508 Kaggle samples exceeding the limit.

6 DISCUSSION

Across five CPFE axes, three independently evaluated models showed simultaneous cross-platform failures, supporting a structural interpretation tied to single-platform proxy-label training and heterogeneous deployment environments. These findings are descriptive baselines, while platform-specific validation and recalibration remain appropriate responses to observed disparities.

  • Structural interpretation: Across all five CPFE axes, degraded discrimination, miscalibration, attribution shift, and prediction-rate disparities co-occurred across both cross-platform test sets in three independently evaluated models.The pattern was not specific to a single evaluated transformer architecture and was consistent with single-platform proxy-label training.
  • Discrimination and calibration: AUC 0.596–0.703 versus 0.983–0.987 indicates severe cross-platform discriminative-utility loss, although performance remains above chance.The clinical AUC ≥0.80 benchmark is referenced only as a conventional comparison because it was established with validated clinical outcomes, not proxy labels.
  • Discrimination and calibration: ECE 0.499–0.542 on Twitter means predicted probabilities deviate substantially from true class probabilities, while F1-macro falls from 0.862–0.883 to 0.284–0.332.The Twitter ECE range corresponds to approximately 50 percentage points on average.
  • Prediction equity: Prior-shift-adjusted DI 0.11–0.29 and EOD 0.753–0.830 on Reddit show substantial prediction disparities beyond prevalence differences alone.The EOD range indicates conditional true-positive-rate drops of 75–83 percentage points relative to within-platform performance; these metrics are diagnostic rather than definitive evidence of algorithmic bias.
  • Scope and limitations: AUC drops exceeding 30% occurred on both platforms across three independently evaluated models, but the results derive from one corpus, three platforms, four models, and proxy emotion labels.They should therefore be treated as descriptive baselines for future cross-platform evaluation rather than validated decision thresholds.

7 LIMITATIONS

The audit’s conclusions are limited by heuristic cross-platform labels, pretraining overlap for two models, and reliance on gradient-based attribution. Further comparisons are needed to determine whether the observed failure extends beyond fine-tuned transformers.

  • Construct validity: Human-annotated clinical labels on Reddit and Twitter are required to disentangle construct mismatch from distributional failure.Emotion-to-clinical mappings are heuristic approximations; Mapping D only provides evidence of robustness to the most salient construct-validity concern.
  • Pretraining contamination: GoEmotions-RoBERTa and Emotion-DistilRoBERTa have pretraining overlap with evaluation domains, making their Reddit performance partially non-independent and upper bounds.
  • Attribution method reliability: Gradient-based saliency is a local linear approximation that can be sensitive to input normalisation and random seed, with limited faithfulness to attribution.
  • Scope of failure: Further comparisons are needed to test whether the observed failure is specific to fine-tuned transformers or general to cross-platform text classification.

8 CONCLUSION

The CPFE framework audited four transformer classifiers across five fairness axes on Reddit and Twitter using remapped mental-health proxy labels. Across evaluated models and seeds, the axes showed co-occurring failure, while calibration improved substantially with temperature scaling and target-domain fine-tuning improved AUC.

  • Conclusion: The five-axis CPFE audit covered discriminative performance, calibration, statistical significance, prediction equity, and attribution stability across Reddit and Twitter.It evaluated four transformer-based classifiers trained on a Kaggle mental health corpus with remapped proxy labels.
  • Conclusion: Across three independently evaluated models and five training seeds, all five CPFE axes revealed co-occurring failure.The evaluated models were BERT, RoBERTa, and Emotion-DistilRoBERTa.
  • Conclusion: 88.0% mean ECE reduction from temperature scaling improved calibration without restoring discrimination.The calibration intervention did not restore discriminative performance.
  • Conclusion: 0.216 mean AUC gain followed single-seed target-domain fine-tuning on the same 10% labelled split.Excluding an outlier pair, the mean AUC gain was 0.197; results were consistent across four label mapping schemes and five independent training seeds.
Loading 2608.26138v1…