PaperlayerView exact revision on arXiv

Source-linked AI summary

Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gate

Yuwen Chiu

arXiv:2609.08161v1cs.CLcs.IR

TL;DR

Early depression detection from sequential Reddit discussions requires deciding not only whether a user is depressed but when to issue an alert under ERDE50’s latency-sensitive costs. Snugi-AI-v2 learns that stopping decision with an MLP policy and sustained confidence gate, achieving strong test performance with rapid evaluation while exposing distribution and representation limitations.

  • Problem

    Prior eRisk Task 2 systems use fixed or heuristic stopping rules rather than learning when to commit under ERDE50’s joint latency and error costs.

  • Method

    The system uses frozen MentalRoBERTa embeddings, an MLP depression classifier, a learned ERDE50 stopping policy, and a sustained confidence gate.

  • Results

    On the official test set, the system achieves F1 = 0.73, Run 3 precision = 0.80, speed = 0.97, and median alert round 8, completing evaluation in 1 hour 26 minutes.

  • Takeaways & Limitations

    The learned policy contributes the largest single improvement, while the sustained N=3 gate adds a further −0.0023 CV ERDE50 improvement.

  • Takeaways & Limitations

    Validation ERDE50 of 0.0267 rises to 0.07–0.09 on the test set, and some depressed users remain undetected because classifier probabilities stay below 0.30.

Abstract

from arXiv · show

We describe the Snugi-AI-v2 submission to eRisk 2026 Task 2, the second edition of contextualized early depression detection from Reddit discussions. Our central contribution is a learned MLP stopping policy trained to directly optimize ERDE50, replacing the fixed and tiered threshold strategies used in all prior eRisk Task 2 submissions. Combined with a sustained confidence gate that commits only after N=3 consecutive rounds of high policy confidence, the system reduces false positives caused by transient emotional posts without sacrificing recall. The pipeline encodes each discussion thread with a frozen MentalRoBERTa model, maps the accumulated representation to a depression probability via an MLP classifier, and delegates the timing decision to the learned policy. Our best run achieves F1 = 0.73 (Run 1) and F_latency = 0.70 (Runs 0 and 3), with a median alert round of 8 out of 500, completing the full evaluation in 1 hour 26 minutes, the fastest among all complete-submission teams. We report a systematic ablation across five runs spanning two encoder variants, four stopping strategies, and three gate values, along with negative results from GRPO policy training, BDI-II post filtering, MentalLongformer encoding, and DeBERTa ensembling. Code: https://github.com/chiuyuwen91/erisk-2026

1. Introduction

Snugi-AI-v2 treats early depression detection as a joint classification-and-timing problem, introducing a learned stopping policy optimized for ERDE50 and a sustained confidence gate. The submission combines these components with systematic ablations across encoders, stopping strategies, and gate values.

  • Early detection matters because identifying at-risk individuals before a crisis can enable timely intervention, while ERDE penalizes both delayed true positives and false positives.
  • Prior Task 2 systems use fixed thresholds, tiered thresholds, or hand-tuned risk windows rather than policies trained to minimize ERDE50.
  • The learned MLP stopping policy maps a five-dimensional probability-trajectory summary to a commitment probability and is trained end-to-end for ERDE50.
  • A sustained confidence gate requires high commitment probability for N=3 consecutive rounds, filtering transient emotional spikes that can cause false alerts.
  • Five runs systematically ablate two encoders, four stopping strategies, and three gate values, with reported best values of F1 = 0.73 and F_latency = 0.70.The full 500-round evaluation took 1 hour 26 minutes, fastest among 16 complete-submission teams.

2. Related Work

The work builds on eRisk’s progression from classical features to transformer encoders and on prior latency-aware risk windows. It extends the risk-window idea by applying a sustained gate to a learned stopping policy rather than a fixed threshold.

  • eRisk research has progressed from hand-crafted features and classical classifiers toward transformer encoders such as MentalRoBERTa, pretrained on 13.6 million mental-health Reddit sentences.
  • Sadeque et al. introduced F_latency and a risk window requiring n consecutive positive predictions before commitment to prevent impulsive alerts.
  • Snugi-AI-v2 extends the risk window by applying a sustained gate to a learned policy trained to output the appropriate commitment probability for each trajectory moment.
  • The contextualized eRisk task provides full discussion threads, while prior high-performing approaches used computationally expensive augmentation, psychiatric-scale screening, and hierarchical attention.

3. Methodology

The methodology processes sequential Reddit discussions with frozen MentalRoBERTa embeddings, an MLP depression classifier, and a learned stopping policy followed by a sustained confidence gate. Incremental pooling supports efficient round-by-round updates while ERDE50 evaluates the trade-off among false negatives, false positives, and alert latency.

  • 3.1. Task and Evaluation: Each round releases one discussion thread, after which the system submits a final-or-reversible binary decision and confidence score evaluated with ERDE50.
  • 3.1. Task and Evaluation: ERDE50 assigns false-negative, latency, or false-positive costs, with cFN = 1.0 and cFP = 0.1296; F_latency equals F1 multiplied by speed.
  • 3.1. Task and Evaluation: Depressed users produce approximately 2.6 times as much cumulative text as controls at every round, motivating stopping decisions that adapt to signal accumulation.
  • 3.1. Task and Evaluation: The latency cost grows after the first 50 rounds, so a dedicated stopping policy must learn when to commit rather than only whether to classify positive.
  • 3.2. Data: The training collection contains 909 subjects, including 102 depressed and 807 control users, with sequences ranging from 1 to 1,279 discussion rounds.
  • 3.3. System Architecture: Frozen MentalRoBERTa-base embeddings are incrementally mean-pooled, adding each new post to a running sum in O(1) time without reprocessing earlier embeddings.
  • 3.3. System Architecture: Mean-pooling avoids cross-attention’s O(N^2) history-dependent interactions and reduces overfitting risk given only 909 training subjects.
  • 3.3. System Architecture: The three-layer classifier maps a 770-dimensional input to a depression probability p_t using batch normalization, ReLU, dropout, and class-weighted BCE loss.

4. Results

The learned stopping policy improves ERDE50 over fixed and tiered threshold strategies while enabling earlier commitment, and its sustained gate and context-dependent variants shape performance across validation and test evaluations.

  • Validation performance: 0.0267 ERDE50 is achieved by Runs 2 and 3 on validation, outperforming fixed threshold at 0.0274 and tiered threshold at 0.0526.The learned policy improves despite slightly lower recall by reducing premature decisions.
  • Cross-validation: 21% lower mean CV ERDE50 is obtained by the learned policy than the fixed-threshold baseline, at 0.029 versus 0.037.The result is reported across 5-fold cross-validation.
  • Validation performance: 16 false positives at median round 6 are achieved by learned-policy Runs 2 and 3, versus 13 false positives at median round 14 for tiered Run 1.Run 1 sacrifices recall despite having fewer false positives.
  • Component ablation: −0.0079 is the largest single component improvement in mean cross-validation ERDE50, contributed by the learned policy.The sustained N=3 gate adds −0.0023 over N=1.
  • Official test results: 0.80 precision is achieved by Run 3 on the official test set, while Run 1 reaches F1 = 0.73 and Runs 0 and 3 reach F_latency = 0.70.Runs 2, 3, and 4 share median alert round 8; the full evaluation took 1 hour 26 minutes.
  • Official test results: 0.85 NDCG@100 places Run 3 fourth among complete-submission teams, while Snugi-AI-v2 is reported as the fastest complete submission at 1 hour 26 minutes.The comparison uses the official 500-writing ranking evaluation.

5. Discussion

The learned stopping policy improves timing-aware detection by using conversation trajectories, while achieving fast alerts and consistent behavior across runs. Performance remains sensitive to distribution shift and unresolved false-positive and false-negative cases.

  • Policy effectiveness: The learned stopping policy outperforms fixed and tiered thresholds on validation and test sets by exploiting trajectory-level depression signals.Its ERDE50 design accounts for false negatives being penalized eight times more heavily than false positives.
  • Limitations: Validation ERDE50 of 0.0267 versus test ERDE50 of 0.07–0.09 reflects both larger test size and distribution differences.The relative ordering of the five runs is preserved on the test set despite the shift in absolute scores.
  • Ablation findings: Context embeddings improve test precision from 0.71 to 0.80 relative to target-only Run 2, with recall changing from 0.66 to 0.65.The validation precision is identical for both runs at 0.515, so the reported benefit appears at test-time scale.
  • Failure cases: Three depressed users are missed across all runs, while nine control users fire early with probabilities above 0.97 because empathetic depression language resembles genuine self-report.The authors identify additional post-intent annotations as necessary to separate these registers.
  • Robustness: Speed remains in [0.95, 0.97] across all five runs, indicating similar stopping points across encoder and strategy variants.The authors describe this stability as useful when per-conversation recalibration is unavailable.
  • Negative results: GRPO training, MentalLongformer encoding, and DeBERTa ensembling did not improve performance, with GRPO collapsing to an all-alert solution.The GRPO result is attributed to the small action space and sparse reward on 909 subjects.
  • Efficiency: Median alert round is 8 of 500 on the official test set, indicating early identification within the first two percent of conversational history.The corresponding latency cost at round 8 is approximately 0.001 under ERDE50.
  • Efficiency: 1 hour 26 minutes is the complete-submission runtime, making the system the fastest among complete-submission teams.The O(1) incremental update supports wall-clock efficiency as conversation history grows.

6. Future Work

Future work targets richer modeling of how depression language evolves across posts and better encoder representations. The authors also propose supplementary signals to address persistent empathetic-language false positives.

  • Sequential representation: Cross-post attention could replace mean-pooling to model relationships among posts over time.A hierarchical attention network is proposed, with efficient approximate attention for online deployment.
  • Encoder adaptation: Fine-tuning the frozen MentalRoBERTa encoder on eRisk data with regularization could yield better representations.The current experiments use MentalRoBERTa only as a frozen feature extractor.
  • False-positive reduction: Supplementary signals about advisory versus personal posting could help distinguish empathetic language from self-report.This direction is motivated by the hard false-positive cluster observed in Figure 4.

7. Conclusions

Snugi-AI-v2 combines a lightweight frozen-encoder pipeline with a learned ERDE50 stopping policy and a three-round confidence gate. On the official test set, it achieves strong detection and speed while documenting several unsuccessful alternatives.

  • System design: The pipeline uses a learned MLP stopping policy optimized for ERDE50 and a sustained gate requiring three consecutive rounds of high confidence.Its lightweight design uses frozen MentalRoBERTa, O(1) incremental mean-pooling, a three-layer classifier, and a five-feature policy.
  • Test results: F1 = 0.73, precision = 0.80, and speed = 0.97 are reported for Run 3 on the official eRisk 2026 Task 2 test set.The complete 500-user evaluation finishes in 1 hour 26 minutes, the fastest complete submission.
  • Ranking evaluation: NDCG@100 = 0.85 at 500 writings places Run 3 fourth among complete-submission teams and remains stable through the full evaluation window.The passage contrasts this stability with two higher-F1 teams whose ranking quality collapses at full conversation length.
  • Ablation and alternatives: The learned stopping policy provides the largest ablation improvement, reducing CV ERDE50 by 0.0079, while the N=3 gate adds 0.0023.The reported negative results cover GRPO, MentalLongformer, DeBERTa ensembling, and BDI-II filtering.

Declaration on Generative AI

The author reports using Claude to assist with debugging, language review, outlining, and visualization, while retaining responsibility for the publication’s content.

  • Use of generative AI: Claude assisted with code debugging, grammar and spelling review, section outlines, and analysis visualizations.The author states that all content was subsequently reviewed and edited.
  • Responsibility: The author takes full responsibility for the publication’s content after using these tools.
Loading 2609.08161v1…