PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Suicide Risk Assessment on Social Media with Multi-Task QLoRA

Xuan Zhong Feng, Geoffrey Martin, Hexin Dong, Yifan Peng

arXiv:2610.00610v2cs.CLcs.LG

TL;DR

Assessing suicide risk from social-media text requires more than a severity label: supporting phrases and expressed risk and protective factors also matter. The paper adapts Qwen2.5 models for three related tasks and selects training and aggregation strategies by output type. Training configurations performed differently across tasks, and the final system achieved a 0.7738 composite leaderboard score.

  • Problem

    Suicide-risk models need to identify supporting language and expressed risk and protective factors as well as estimate a risk category.

    Source: §1 · pp. 1–2

  • Method

    The system adapts Qwen2.5 models with QLoRA and uses task-specific training configurations and aggregation for risk classification, evidence extraction, and factor identification.

    Source: §3.1 · p. 4, §7 · p. 11

  • Results

    Three-task training performed best for Task 1a, joint Tasks 1a and 1b training for Task 1b, and task-specific training for Task 2; the final system scored 0.7738 composite on the official leaderboard.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The experiments support tailoring training and ensemble design to each task's output structure, while training-exposure differences mean the observed associations are not causal.

    Source: §7 · p. 11

    The dataset is small and from a single Reddit community, limiting evidence for generalization across platforms, populations, time periods, and clinical settings; post-level modeling also misses longitudinal risk changes.

    Source: §6 · p. 11

Abstract

from arXiv · show

Explainable suicide-risk assessment requires models not only to estimate risk severity, but also to identify supporting language and the risk and protective factors expressed in a post. We present our system for the IEEE BigData 2026 Cup on Explainable Suicide Risk Assessment on Social Media, which addresses three tasks: risk-level classification, evidence phrase extraction, and multi-label factor identification. Our approach adapts Qwen2.5-Instruct models using quantized low-rank adaptation (QLoRA) and an answer-masked causal language-model objective. We jointly train across all three tasks for risk classification, jointly train on Tasks 1a and 1b for evidence extraction, and adapt Task 2 separately for factor identification. We also tailor aggregation to each output: we average risk-level probabilities from the 32B and 72B models, combine evidence phrases through cross-fold consensus, and calibrate factor-specific decisions through rate matching based on out-of-fold operating points. On the official leaderboard, the final system achieved a composite score of 0.7738, with 0.8089 on Task 1 and 0.6919 on Task 2. Across the evaluated configurations, three-task training performed best for Task 1a, joint training on Tasks 1a and 1b performed best for Task 1b, and task-specific training performed best for Task 2. Probability averaging further improved Task 1a when component models had complementary errors. These findings highlight the value of tailoring both training objectives and aggregation strategies to the output structure of each task within a unified language-model framework.

1. Introduction

Explainable suicide-risk assessment must identify more than a risk category: it should show the language supporting the assessment and the risk and protective factors expressed. The paper addresses three related tasks with a definition-aware language-model framework and reports its official leaderboard results.

1. Introduction

  • Suicide-risk assessment from social-media text requires identifying both the supporting language and the risk and protective factors, not merely predicting a risk category.The introduction frames suicide prevention as an important public-health concern and social-media language as potentially complementary to traditional care settings.Source: §1 · p. 1
  • The three related tasks—risk classification, evidence-span extraction, and multi-label factor identification—differ in objectives and output structures and face informal language, limited annotations, and severe class imbalance.Source: §1 · p. 2
  • The submission uses natural-language instructions and structured outputs to share relevant information across the three subtasks while preserving distinct objectives and selecting training configurations by output.The shared task formalizes the subtasks.Source: §1 · p. 2
  • 0.7738 composite score, with 0.8089 on Task 1 and 0.6919 on Task 2, was achieved by the final system on the official leaderboard.The system uses aggregation strategies tailored to each task's output structure.Source: §1 · p. 2

2. Task and Dataset

The shared task evaluates risk-level classification, supporting-phrase extraction, and identification of applicable suicide-related factors from Reddit posts. Its labeled data are drawn from one Reddit community and contain imbalanced risk and factor labels.

2.1. Task Definition

  • Task 1a classifies four risk levels, Task 1b extracts supporting phrases, and Task 2 identifies applicable labels from a 24-factor taxonomy.The tasks are evaluated with weighted F1 for risk, Phrase F1 for evidence, and macro F1 over factors.Source: §2.1 · p. 2

2.2. Dataset and Annotation

  • The labeled set contains 1,635 posts from 153 users, including 114 Attempt posts (7.0%); factor-positive counts range from 8 to 745 posts.Annotations include 1,833 evidence phrases and 4,772 factor assignments; the training-set distributions are shown in Table 1.Source: §2.2 · p. 3, Table 1 · p. 3
  • The system architecture uses probability averaging for Task 1a, cross-fold phrase consensus for Task 1b, and rate-matched operating points for Task 2.Source: Figure 1 · p. 4

3. Methodology

The methodology uses a shared instruction-based QLoRA framework for three related tasks, with separate training configurations and aggregation procedures tailored to their output structures. Risk labels are ensembled, evidence phrases are combined by cross-fold consensus, and factor decisions are calibrated separately.

3.1. System Overview

  • The final system combines all-three-task 32B and 72B models for risk classification, a 72B model jointly trained on Tasks 1a and 1b for evidence spans, and a Task 2-only 32B model for factor predictions.Risk probabilities are averaged across models, evidence phrases use cross-fold consensus, and factor decisions use rate-matched operating points.Source: §3.1 · p. 4

3.2. Definition-Aware Multi-Task Adaptation

  • Across tasks, Qwen2.5-Instruct models are adapted with QLoRA and task-specific instructions, while the loss is restricted to answer tokens rather than prompts or fixed output scaffolding.Joint training supplies one example per included task for each post, enabling a shared representation.Source: §3.2 · pp. 4–5

3.3. Risk-Level Classification (Task 1a)

  • Task 1a constrains prediction to four risk labels and averages normalized class probabilities across five fold-specific adapters at each of two model sizes.The ablations compare Task 1a-only, Tasks 1a and 1b jointly, and three-task training, alongside a MentalRoBERTa baseline.Source: §3.3 · p. 5

3.4. Evidence Phrase Extraction (Task 1b)

  • Task 1b prompts extractive generation of the shortest exact-wording spans supporting the risk judgment, then retains phrases produced by at least two of five fold-specific adapters.The two-of-five threshold yielded 1.146 phrases per post, closer to the OOF rate of 0.985 than the five-adapter union rate of 1.521.Source: §3.4 · pp. 5–6

3.5. Factor Identification (Task 2)

  • Task 2 presents the factor codebook and 24 factor names, evaluates them sequentially with cached shared context, and allows later decisions to condition on earlier predictions.A separate OOF F1-maximizing threshold per factor determines its positive rate; quantile cutoffs on ensembled test probabilities preserve that operating point despite a changed score scale.Source: §3.5 · p. 6
  • Task 2 ablations compare LLM adapters with a MentalRoBERTa 24-output baseline and per-factor late fusion using three GPT-5.4 prompt families.The encoder uses factor-wise asymmetric loss; late fusion selects each factor's prompt family by validation-fold F1.Source: §3.5 · pp. 6–7

4. Experimental Setup

The experiments used Qwen-family and encoder configurations with defined training and inference settings. Evaluation used five-fold cross-validation, with user-grouped folds and qualifications on how some reported estimates relate to final systems.

4.1. Training Configuration

  • Models used 4-bit NormalFloat quantization, double quantization, bfloat16 computation, and LoRA updates across attention and feed-forward modules.Training used one A100-SXM4-80GB GPU per model; optimization used AdamW, two epochs, and an effective batch size of 8.Source: §4.1 · p. 7
  • Task 1b used greedy decoding capped at 128 new tokens, while Tasks 1a and 2 scored constrained answer positions without free-text decoding.Task 2-only training also clipped gradient norms at 1.0.Source: §4.1 · p. 7

4.2. Cross-Validation and Evaluation Protocol

  • The five-fold protocol pooled out-of-fold predictions, with each table caption specifying the task-specific metric.Posts were grouped by user and stratified by risk level; no user appeared in more than one fold.Source: §4.2 · p. 7, Table 2 · p. 8
  • Task 1b validation scores used one held-out adapter per post, so they did not directly evaluate the final 2-of-5 consensus system.GPT-5.4 prompt mapping was selected using the same five validation folds, making its pooled score a non-nested estimate not directly comparable with fully held-out model estimates.Source: §4.2 · pp. 7–8

5. Results

The final system scored 0.7738 overall on the official leaderboard, while the best training and aggregation choices differed across risk classification, evidence extraction, and factor identification.

5.1. Official Leaderboard Performance

  • 0.7738 composite score, with 0.8089 on Task 1 and 0.6919 on Task 2, was achieved by the final official leaderboard system.It averaged risk probabilities, used 2-of-5 evidence-phrase consensus, and applied factor-specific rate-matched decisions.Source: §5.1 · p. 8

5.2. Task 1a: Risk-Level Classification

  • 0.8395 weighted F1 was achieved on Task 1a by averaging the three-task Qwen2.5-32B and Qwen2.5-72B class-probability vectors.The three-task models scored 0.8259 and 0.8285 weighted F1, respectively; their error correlation was 0.68, and their soft-voting ensemble reached 0.601 accuracy on 158 disagreement posts.Source: §5.2 · p. 8

5.3. Task 1b: Evidence Phrase Extraction

  • 0.7874 Phrase F1 was the highest single-adapter pooled out-of-fold result, from Qwen2.5-72B jointly trained on Tasks {1a, 1b}.These validation scores use one held-out adapter per post, rather than the final 2-of-5 consensus.Source: §5.3 · p. 9, Table 3 · p. 9

5.4. Task 2: Suicide-Factor Identification

  • 0.7207 macro F1 was achieved by Task 2-only QLoRA with Qwen2.5-72B, exceeding the 0.7000 achieved with Qwen2.5-32B.QLoRA gains over corresponding zero-shot baselines ranged from 0.0616–0.0716; Task 2-only models outperformed three-task QLoRA for both sizes.Source: §5.4 · p. 10
  • 0.7309 was the best Task 2 out-of-fold macro F1, from averaging Qwen2.5-32B and Qwen2.5-72B probabilities, but the final submission retained Qwen2.5-32B.Its leaderboard score was 0.6919, versus 0.6912 for Qwen2.5-72B; Task 2 results are pooled five-fold results over 1,635 posts.Source: Table 4 · p. 9, §5.4 · p. 10

5.5. Cross-Task Findings

  • The best training-task mixture depended on target task and backbone, so the authors selected it independently for each output.Probability averaging helped when models had complementary errors, but out-of-fold gains did not always translate to stronger leaderboard performance, where macro-F1 was sensitive to threshold selection and rare factors.Source: §5.5 · p. 10

6. Discussion

The findings associate training benefits with target density and complexity, while ensemble gains depend on compatible output representations and complementary predictions. The authors caution that differing optimization exposure, rare factors, and a small single-community dataset limit interpretation and generalization.

Training supervision

  • Task 1a benefited from richer auxiliary supervision, Task 1b improved with Task 1a supervision, and Task 2 performed best with dedicated adaptation.Adding Task 2 further improved the 32B Task 1b model but reduced Phrase F1 for the 72B model; differing optimizer updates prevent causal attribution.Source: §6 · pp. 10–11

Ensemble effectiveness

  • 0.7309 nested macro-F1 was reached by averaging Task 2 model probabilities, versus 0.7000 and 0.7207 for the Qwen2.5-32B and Qwen2.5-72B outputs, respectively.For Task 1a, averaging complementary 32B and 72B errors increased weighted F1 to 0.8395.Source: §6 · p. 11

Validation and adaptation

  • Higher pooled OOF macro-F1 did not always predict leaderboard success for Task 2, where larger models and cross-model ensembles did not surpass Qwen2.5-32B.Rare factors make threshold estimation and macro-F1 variable; QLoRA beat full fine-tuning only in the tested Qwen2.5-32B configuration, not establishing a general advantage.Source: §6 · p. 11

Limitations

  • The dataset has 153 labeled users from one Reddit community, including 114 Attempt posts and as few as eight positives for some factors, limiting generalization across settings.Post-level modeling also misses longitudinal risk changes; the authors call for external and longitudinal evaluation, improved rare-factor calibration, update-matched comparisons, repeated seeds, and broader adaptation comparisons.Source: §6 · p. 11

7. Conclusion

The conclusion presents a definition-aware instruction framework with distinct training and aggregation choices for risk classification, evidence extraction, and factor identification. It reports that related supervision was associated with gains for lower-density targets, while factor identification favored separate adaptation, and calls for cautious evaluation before broader use.

Conclusion

  • The system combines joint adaptation and probability ensembling for risk classification, cross-fold phrase consensus for evidence extraction, and task-specific adaptation with factor-wise decision rules for factor identification.The framework uses task-specific instructions and output formats across the three tasks.Source: §7 · p. 11
  • Related supervision was associated with better performance on lower-density targets, whereas the 24-decision factor-identification task benefited from separate adaptation.Because training exposure differed across configurations, the reported associations are not causal evidence.Source: §7 · p. 11
  • The authors recommend external and longitudinal evaluation and state that use beyond research would require clinical validation, strong privacy safeguards, and human oversight.Future work should also improve estimation for rare factors.Source: §7 · p. 11

A. Prompt Templates

The appendix documents task-specific prompt formats for risk classification, evidence extraction, and factor identification, alongside three GPT-5.4 factor-prompt variants. The templates define output constraints and evidence thresholds, with more detailed coding guidance for selected complex factors.

A.1. Risk-Level Classification (Task 1a)

  • Task 1a prompts assign one of four ordered risk levels: Indicator, Ideation, Behavior, or Attempt.The prompt ends with a forced answer slot containing the risk-level label.Source: §A.1 · p. 14

A.2. Evidence Phrase Extraction (Task 1b)

  • Task 1b requests the shortest verbatim post spans supporting the risk judgment, one per line, or NONE when no evidence is present.The response must not paraphrase, explain, or number the spans.Source: §A.2 · p. 14

A.3. Suicide-Factor Identification (Task 2)

  • Task 2 presents definitions for 24 factors and requires an ordered Yes/No response for each, with Yes reserved for clear evidence under the definition.Each factor answer is a single token; the zero-shot control uses the same prompt and answer grid without Task 2 parameter adaptation.Source: §A.3 · pp. 14–16

A.4. GPT-5.4 Prompt-Ensemble Variants (Task 2)

  • The GPT-5.4 variants share a user message and JSON response schema, with factor-definition blocks inserted programmatically and NVDRS guidance used in the recommended variant.The appendix also describes replacing <POST> with the post and using gold outputs during Qwen training.Source: §A.4 · p. 16

A.4.1. Concise NVDRS-Essence Prompt

  • The concise NVDRS-essence prompt uses factor definitions and selects categories that are directly stated or strongly implied, avoiding broad labels from general distress alone.It requires exact category names and JSON output.Source: §A.4.1 · p. 16

A.4.2. NVDRS-Recommended Prompt

  • The recommended prompt uses concise definitions for 17 simple factors and longer NVDRS-style guidance for seven complex labels, without forcing forensic details absent from social-media text.For complex labels, a supported mapped NVDRS circumstance suffices as evidence for the corresponding factor.Source: §A.4.2 · p. 17

A.4.3. Factor-Specific Strictness-Control Prompt

  • The strictness-control prompt applies factor-specific strict, neutral, or loose evidence thresholds rather than one global threshold.Its instructions range from concrete evidence only to coding plausible contextual suggestions.Source: §A.4.3 · p. 17
Loading 2610.00610v2…