PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient or Powerful? Trade-offs Between Machine Learning and Deep Learning for Mental Illness Detection on Social Media

Zhanyi Ding, Zhongyan Wang, Yeyubei Zhang, Yuchen Cao, Yunchong Liu, Xiaorui Shen, Yexin Tian, Jianglai Dai

arXiv:2503.01082v1cs.CL

TL;DR

Mental-health detection from social-media text requires choosing between ML and DL under trade-offs in performance, interpretability, and computational efficiency. This study empirically compares these approaches across binary and multiclass tasks using a public dataset, finding that model choice should reflect the evaluation setting and interpretability requirements.

  • Problem

    The paper addresses the need for empirical comparisons of ML and DL performance and interpretability across mental-health classification tasks.

  • Method

    The study evaluates ML and DL models for binary and multiclass mental-health classification using a public Kaggle dataset, weighted evaluation metrics, and feature-importance measures.

  • Results

    The study reports model performance using weighted F1 and AUROC while comparing interpretability through model-specific feature-importance measures.

  • Takeaways & Limitations

    Model selection should consider the trade-offs among accuracy, interpretability, and computational efficiency in social-media mental-health classification.

  • Takeaways & Limitations

    Model generalizability is constrained by dataset bias, preprocessing inconsistencies, and reliance on imbalanced training data.

Abstract

from arXiv · show

Social media platforms provide valuable insights into mental health trends by capturing user-generated discussions on conditions such as depression, anxiety, and suicidal ideation. Machine learning (ML) and deep learning (DL) models have been increasingly applied to classify mental health conditions from textual data, but selecting the most effective model involves trade-offs in accuracy, interpretability, and computational efficiency. This study evaluates multiple ML models, including logistic regression, random forest, and LightGBM, alongside deep learning architectures such as ALBERT and Gated Recurrent Units (GRUs), for both binary and multi-class classification of mental health conditions. Our findings indicate that ML and DL models achieve comparable classification performance on medium-sized datasets, with ML models offering greater interpretability through variable importance scores, while DL models are more robust to complex linguistic patterns. Additionally, ML models require explicit feature engineering, whereas DL models learn hierarchical representations directly from text. Logistic regression provides the advantage of capturing both positive and negative associations between features and mental health conditions, whereas tree-based models prioritize decision-making power through split-based feature selection. This study offers empirical insights into the advantages and limitations of different modeling approaches and provides recommendations for selecting appropriate methods based on dataset size, interpretability needs, and computational constraints.

Efficient or Powerful? Trade-offs Between Machine Learning and Deep

This section frames mental-health detection on social media as a comparison between ML and DL under trade-offs involving performance, interpretability, and computational efficiency. It motivates an empirical evaluation across binary and multiclass tasks using a public dataset and weighted evaluation metrics.

  • Motivation: Social media provides large-scale, real-time textual signals for monitoring mental-health conditions and distress.These data motivate ML and DL approaches to automated detection.
  • Challenges: Dataset bias, preprocessing inconsistencies, class imbalance, and informal or context-dependent language complicate generalizable detection.The cited challenges include sarcasm, slang, and meanings that depend on context.
  • Research gap: The central problem is balancing model performance against interpretability when selecting ML or DL methods.Traditional ML offers feature-importance-based transparency, whereas DL captures linguistic patterns but can function as a black box.
  • Study design: The study systematically compares ML and DL models across binary and multiclass mental-health classification tasks.It uses a publicly available Kaggle dataset containing multiple mental-health conditions.
  • Evaluation: Evaluation uses weighted F1 and AUROC, while interpretability is assessed through coefficients, impurity reduction, and gain-based feature rankings.The measures are intended to account for class imbalance and expose influential predictors.
  • Contribution: The paper examines accuracy, interpretability, and computational efficiency to inform model selection for social-media mental-health classification.Its contribution is framed as empirical guidance rather than a single universally best model.

Method

The method combines social-media data preparation with shared preprocessing, model-specific feature representations, and class-structured evaluation. Traditional ML uses engineered TF-IDF features, whereas DL can operate on raw sequences or dense embeddings.

  • Implementation: Experiments use Python libraries for preprocessing, traditional ML, DL, and pretrained language models, with training performed on Google Colab using an NVIDIA T4 GPU.The workflow includes systematic hyperparameter tuning and performance evaluation.
  • Data collection: The study collects mental-health text from platforms including Reddit, Twitter, and Facebook, then removes duplicates, spam, irrelevant content, and personal identifiers.Mental-health labels are standardized during acquisition.
  • Labeling: The dataset supports binary classification by grouping non-Normal labels together and multiclass classification using six original categories.The multiclass labels are Normal, Depression, Suicidal, Anxiety, Stress, and Personality Disorder.
  • Representations: Traditional ML receives structured numerical features, while DL processes raw text sequences or dense vector embeddings.This difference determines whether feature extraction is explicit or integrated into the model.
  • Representations: ML feature extraction uses TF-IDF with unigrams and bigrams, limiting the representation to 1,000 features for efficiency and overfitting control.TF-IDF emphasizes informative terms while downweighting common ones.

Model Development

The model-development pipeline compares linear, kernel, tree-ensemble, boosting, and neural approaches for binary and multiclass prediction. It also evaluates feature-based interpretability and addresses class imbalance through weighted metrics and class weighting.

  • Model scope: The study develops models spanning simple linear classifiers, nonlinear classifiers, ensemble trees, gradient boosting, and deep-learning architectures.Models are selected to capture different relationships in textual data.
  • Logistic Regression: Logistic regression estimates mental-health outcome probabilities from weighted feature combinations and supports binary and multiclass prediction.Regularization and grid search tune the model using weighted F1.
  • Support Vector Machines: SVM experiments evaluate linear and RBF-kernel models with grid search over regularization, class weighting, and γ.The multiclass implementation uses one-versus-one classification with majority voting.
  • Tree-Based Models: Random Forests aggregate bootstrap-trained trees and tune estimator count, depth, and split criteria using weighted F1.Feature-importance analysis supplies information about variables influencing decisions.
  • Tree-Based Models: LightGBM uses leaf-wise growth and histogram-based binning to target efficient, scalable boosting for high-dimensional data.Its grid search tunes boosting iterations, learning rate, tree depth, leaves, and child-sample requirements.
  • Tree-Based Models: Tree-based models distinguish Normal from Abnormal cases and predict multiple mental-health categories while exposing influential predictors through built-in importance measures.The reported multiclass categories include Normal, Depression, Anxiety, and Personality Disorder.

A Lite Version of Bidirectional Encoder Representations from Transformers (ALBERT)

ALBERT is presented as an optimized BERT variant that reduces parameterization while retaining language-modeling capability. The study fine-tunes it for binary and multiclass mental-health classification with randomized hyperparameter search.

  • Architecture: ALBERT is an optimized variant of BERT designed to improve computational efficiency while preserving NLP performance.Its refinements include parameter sharing and factorized embedding parameterization.
  • Architecture: Sentence Order Prediction is introduced as an auxiliary pretraining task to improve sentence-level coherence.This is part of ALBERT’s architectural refinements.
  • Tasks: The study fine-tunes pretrained Albert-base-v2 for both binary and multiclass mental-health classification.The binary task separates Normal and Abnormal statuses, while the multiclass task predicts mental-health categories.
  • Training: ALBERT hyperparameters are tuned with random search over 10 iterations, including learning rate, dropout rate, and training epochs.Weighted F1 is the primary performance metric, and multiclass training uses weighted cross-entropy for class imbalance.
  • Rationale: ALBERT is described as capturing long-range textual dependencies while providing computational advantages.The model is positioned as suitable for large-scale text classification applications.

Gated Recurrent Units (GRUs)

GRUs model sequential text dependencies through an embedding layer, recurrent processing, and classification layers. They offer efficient text classification and capture long-range dependencies, but are less interpretable than tree-based models.

  • GRUs are recurrent neural-network variants designed to model sequential dependencies in natural language.
  • GRUs simplify LSTM gating by merging the forget and input gates into one update gate, reducing parameters and computational cost.
  • The study applied GRUs to binary Normal-versus-Abnormal classification and multi-class mental health classification.
  • The GRU architecture uses embeddings, a GRU layer whose final hidden state feeds a classifier, and a fully connected output layer.
  • Dropout, weighted cross-entropy, random-search tuning, and weighted F1 model selection addressed overfitting, class imbalance, and model configuration.
  • GRUs captured sequential linguistic patterns and long-range dependencies efficiently, although they were less interpretable than tree-based models.
  • Precision and recall involve competing risks: excessive recall can increase false positives, whereas precision-focused models may miss true positives.
  • F1 balances precision and recall, while AUROC evaluates discrimination across thresholds but may be less informative under severe class imbalance.

Results

The results section evaluates dataset composition and model-development procedures for binary and multi-class mental health classification. The dataset is substantially imbalanced, motivating metrics that account for unequal class representation.

  • Evaluation procedure: The analysis tuned model hyperparameters and compared binary and multi-class performance using F1 scores and AUROC.
  • Dataset overview: The dataset contains 52,681 unique textual statements annotated with mental health status labels.
  • Dataset overview: The dataset is heavily imbalanced, with some mental health categories represented substantially more frequently than others.
  • Class distribution: Normal and Depression were the largest listed categories, comprising 31.02% and 29.24% of statements, respectively.
  • Class distribution: Suicidal statements comprised 20.22%, while Anxiety comprised 7.29% of the dataset.
  • Task construction: For binary classification, all listed mental health conditions were combined into Abnormal, while Normal remained unchanged.
  • Task construction: The binary transformation produced 16,343 Normal statements (31.02%) and 36,338 Abnormal statements (68.98%).

Computational Efficiency

Training time varied substantially across architectures, with computational cost driven primarily by model architecture rather than the number of classes. Deep learning models generally required more time than machine learning models.

  • Training time varied significantly across binary and multi-class tasks, with computational cost driven primarily by model architecture rather than class count.
  • Machine learning models: SVM required exceptionally long training times compared with Logistic Regression, Random Forest, and LightGBM for both tasks.
  • Deep learning models: ALBERT and GRU consistently required more training time than machine learning models because of their higher computational complexity.
  • Task comparison: Machine learning models took longer to train for multi-class classification than for binary classification.
  • Task comparison: Deep learning models showed no notable training-time difference between binary and multi-class classification.

Performance Metrics

Machine learning and deep learning models achieved strong classification performance, with deep learning generally ahead and differences remaining small. Multi-class tasks lowered F1 scores more consistently than AUROC, reflecting greater classification complexity and class-specific errors.

  • Binary classification: Binary-model weighted F1 scores ranged from 0.9345 for Logistic Regression to 0.9576 for ALBERT, while AUROC values ranged from 0.92 to 0.95.
  • Cross-model comparison: Deep learning models generally outperformed machine learning models, although the performance differences were small.
  • Multi-class classification: For multi-class classification, ALBERT achieved the highest weighted F1 score of 0.7841, while ALBERT, LightGBM, and GRU shared AUROC of 0.97.
  • Task comparison: Multi-class classification produced lower F1 scores than binary classification across all models because seven categories require multiple decision boundaries.
  • Metric interpretation: AUROC remained consistently high across binary and multi-class tasks, indicating strong class discrimination despite added complexity.
  • Error analysis: In the illustrative LightGBM confusion matrix, Suicidal and Depression were frequently misclassified as Stress and Normal, reducing precision and recall.
  • Metric interpretation: The illustrative LightGBM ROC analysis reported AUC above 0.90 for most classes and micro-average AUROC of 0.97, excepting Depression and Suicidal from that threshold pattern.

Error Analyses

Errors were concentrated in distinguishing depression from overlapping or emotionally similar classes, with underrepresented categories and inconsistent labels adding difficulty. The LightGBM results show depression had the weakest multiclass discrimination.

  • Class overlap: Depression and Personality Disorder were frequently confused in both directions.The passages attribute this pattern to significant overlap between the classes.
  • Class overlap: Suicidal was frequently misclassified as Depression, likely because of overlapping linguistic patterns.
  • Depression discrimination: Depression had the lowest LightGBM AUC, at 0.90, among all classes.Its AUC dropped further for other models, indicating persistent difficulty distinguishing depression from other classes.
  • Data limitations: Inconsistent labeling across data sources and ambiguous class boundaries further complicated classification.The dataset combined data from multiple sources whose labels may not have remained consistent when unified.
  • Data limitations: Bipolar and Personality Disorder were underrepresented, exacerbating multiclass misclassification and lowering some class-level F1 scores.

Model Interpretability

The section compares how traditional machine-learning models expose feature importance with how deep-learning models derive representations from raw text. Logistic regression offers directional interpretability, whereas tree-based criteria and nonlinear transformations make direct feature contributions harder to interpret.

  • Deep learning: Deep-learning models extract representations from raw text and do not produce traditional variable-importance scores.The passages characterize these models as learning hierarchical patterns across sentences and contexts, making them black-box models.
  • Tree-based models: Random forest uses Gini impurity reduction, while LightGBM ranks features by information gain from node splits.
  • Logistic regression: Logistic regression provides the clearest feature interpretation because coefficients retain both magnitude and sign.Positive and negative coefficients distinguish terms associated with higher or lower likelihood of the outcome.
  • Cross-model patterns: Top features overlap strongly across machine-learning models, although importance rankings vary with each model’s criterion.All models consistently highlighted depression-related language as a strong predictor.
  • Binary classification: In binary classification, ‘depression’ was the most predictive feature across methods, alongside emotional-distress terms such as ‘feel,’ ‘want,’ and ‘anxiety.’Logistic regression emphasized ‘restless’ and ‘suicidal,’ whereas tree-based models emphasized ‘die,’ ‘kill,’ and ‘suicide.’
  • Multiclass classification: Multiclass rankings shifted toward condition-specific terms: tree-based models emphasized ‘bipolar’ and ‘anxiety,’ while logistic regression emphasized ‘restless’ and ‘nervous.’

Discussion

The discussion frames model selection as a trade-off among dataset size, interpretability, classification performance, and computational efficiency. ML models are practical for smaller datasets and constrained resources, while DL models become more suitable as data volume and linguistic complexity increase.

  • Study scope: The study compares ML and DL models for mental health classification across accuracy, interpretability, and computational efficiency.The comparison includes traditional ML and DL approaches for social-media-based mental health detection.
  • Dataset size: ML models remain effective for small-to-medium datasets because feature engineering supports a more controlled and interpretable learning process.This benefit requires additional preprocessing and careful input-variable selection.
  • Dataset size: DL models can extract complex linguistic patterns without extensive feature engineering, but their advantage is less pronounced with limited training samples.The discussion associates DL models with large-scale training data and large-volume mental health detection tasks.
  • Computational efficiency: ML models consistently require less computational time, whereas DL models show competitive classification performance but substantially longer training times.The efficiency difference matters particularly when computing resources are constrained or deployment must scale.
  • Interpretability: Logistic regression offers the clearest interpretability by distinguishing positive and negative feature associations, while tree-based and DL models provide less direct explanations.Tree-based models rank features by contribution to accuracy, and DL models lack explicit feature-importance scores.
  • Limitations: Model assumptions, inconsistent class labels, uncertain annotation quality, and ethical concerns constrain reliable mental health classification from social-media data.The discussion highlights linearity assumptions, ambiguous multi-class boundaries, subjective labels, and the need for careful handling of sensitive information.
Loading 2503.01082v1…