PaperlayerView exact revision on arXiv

Source-linked AI summary

Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessment

Shlok Shelat, Shrey Salvi, Souvik Roy, Manas Gaur, Amit Sheth

arXiv:2609.07766v1cs.CLcs.AI

TL;DR

Small, high-stakes suicide-risk assessment from social media requires multiple outputs, yet common modeling techniques had not been audited in the small-data, coupled-output regime. The paper tests those techniques under controlled conditions and selects components according to the task knowledge they can use. Only 5 of 31 comparisons held, supporting task-conditioned technique selection within this task setting.

  • Problem

    The standard NLP playbook had not been audited on small clinical text tasks with coupled outputs and macro-averaged scores over highly unequal label counts.

    Source: §I · pp. 1–2

  • Method

    The authors conduct controlled comparisons of standard techniques and use a four-step procedure to identify task knowledge, test its informativeness, and retain components only when comparisons support them.

    Source: §IV · p. 7, §VI · p. 12

  • Results

    5 of 31 pre-specified comparisons held, while 26 did not; the authors report that most effects were smaller than this corpus could resolve and 6 components measurably worsened results.

    Source: §VIII · p. 17

  • Takeaways & Limitations

    For this task shape, the authors conclude that task knowledge can guide which components to keep and which machinery to remove.

    Source: §VIII · p. 17, §IX · p. 18

    The audit covers one English-language platform, one codebook, and one metric, with no prospective clinical evaluation, demographic performance analysis, or independent-dataset replication.

    Source: §IX · p. 18

Abstract

from arXiv · show

Assessing suicide risk from social media text is a small-data, high-stakes setting requiring not only severity prediction but also supporting evidence and clinically relevant risk and protective factors. Yet common NLP techniques, including model scaling, synthetic data, loss reweighting, ensembling, and threshold tuning, are often applied without testing whether their gains hold up under severe class imbalance, coupled outputs, and limited author-level data. We study 1,635 clinician-annotated posts and audit 31 pre-specified techniques from 7 methodological families through roughly 300 controlled experiments on author-disjoint partitions. We found no prior audit of this playbook in this regime. The findings guide a task-grounded system for three outputs: 4-level suicide risk, evidence spans, and 24 clinical risk and protective factors. Only 5 of 31 comparisons produced reliable gains. We reformulate factor prediction as entailment between each post and its codebook definitions, using an architecturally diverse ensemble with class-balanced training and score rescaling. Risk predictions condition a 7-model evidence tagger ensemble; evidence restricts symbolic risk rules; and a difficult risk class is routed separately. The factor predictor remains independent because risk evidence provides no additional factor signal. We also correct a mismatch between validation scores used for threshold fitting and test-time ensemble scores through deployment-consistent calibration, yielding the largest improvement to the factor system. The final system achieves 0.8203 for risk, 0.7953 for evidence, and 0.7045 macro-F1 for factors, with a 0.7781 composite, ranking third among 53 teams. We call the underlying principle task-conditioned technique selection: retain techniques only when task-specific knowledge, structure, or empirical evidence justifies them.

I. Introduction

The paper audits whether standard modeling techniques hold in a small, multi-output clinical text task, then argues for selecting components according to the task knowledge they can use. The audit motivates a task-conditioned design rather than accumulating techniques by default.

I. Introduction

  • Only 5 of 31 pre-specified comparisons held, while 26 produced no effect distinguishable from run-to-run variation.The study tested roughly 300 controlled experiments across 7 technique families on author-disjoint partitions.Source: §I · pp. 1, 3, Figure 1 · p. 3
  • The audit targets a task with 1,635 posts, 3 coupled outputs, and a macro-average that weights categories with 8 and 745 positive examples equally.The authors argue these conditions leave practitioners with little data to validate borrowed techniques individually.Source: §I · p. 2
  • Definition conditioning raised the factor score from 0.3795 to 0.6172, but scored 0.4830 against 0.7787 for ordinary evidence tagging.The codebook defines the factors, whereas it does not specify which words justify a risk judgment.Source: §I · p. 2
  • 0.0137 in factor macro-F1 and +0.0041 on the composite came from correcting threshold calibration to match the deployed ensemble’s score distribution.The correction fit no new parameters, and the held-out protocol could not reveal the mismatch because both threshold fitting and comparison used single-fold predictions.Source: §I · p. 3
  • The five techniques that held each exploited task-specific knowledge, including three kinds available before training and two discovered through measurement.The authors test this principle in matched comparisons where the relevant knowledge is present and absent.Source: §I · pp. 3–4
  • The paper’s proposed contribution is to use task knowledge to decide which components to keep, removing modeling choices that the task cannot support.The authors contrast this with the familiar playbook of scaling models, generating data, adapting to domains, reweighting losses, ensembling, and tuning thresholds.Source: §I · pp. 2, 4

II. Related Work

Prior work has studied suicide-risk systems and audits of standard techniques, but the paper identifies a gap at their intersection: a controlled audit for small, multi-output clinical text tasks with macro-averaged scores. It also situates definition conditioning as an established approach with prior evidence in scarce-data settings.

II. Related Work

  • The two literatures remain separate: suicide-risk systems commonly take general-purpose machinery for granted, while audits of that machinery had not tested this task regime.Cross-system shared-task tables confound technique with team, model, budget, and seed, unlike a controlled protocol.Source: §II · pp. 4–5
  • Suicide-risk systems moved from graded risk labels to evidence spans and 24 clinical risk and protective factors, while prior shared-task pseudo-labeling did not show a distinguishable effect here.The task builds on earlier resources and competition editions, which established the outputs as actionable additions to a binary risk flag.Source: §II · p. 4
  • Earlier audits found that many proposed techniques fail under controlled re-testing, but none covered small clinical text tasks with coupled outputs and steeply imbalanced macro-averaged labels.The nearest cited studies address single-label mental-health classification or large-data, single-output settings.Source: §II · pp. 4–5
  • Definition conditioning is an established method, and prior work found gains on scarce-labeled-data tasks, a range that includes this corpus of 1,635 posts.The cited evidence spans 8 tasks with 100 to 2,500 labeled texts and reports gains of 10 to 18 points over classical models without transfer learning.Source: §II · p. 5

III. What This Task Supplies, and What It Withholds

The task combines three interdependent outputs with two kinds of task knowledge: definitions supplied by the clinical codebook and empirical properties of the data. Its author-level splits and distinct scoring rules reflect the corpus’s small, dependent, imbalanced structure.

III. What This Task Supplies, and What It Withholds

  • Task knowledge comprises codebook definitions available before training and measured data properties that must be discovered empirically.The distinction matters because the first kind can guide technique choice without additional experiments, whereas the second requires measurement.Source: §III · p. 5
  • A system must predict four ordered risk levels, verbatim supporting evidence, and 24 risk or protective factors read independently from risk.There are 19 risk factors and five protective factors; posts may show multiple factors or none.Source: §III · p. 5
  • The 1,635 annotated posts come from 153 authors, so validation and five-fold cross-validation split by author rather than post.Posts by one author resemble one another, and author-only knowledge raises odds of rare-category labels by 6.2 times versus 1.1 for common categories.Source: §III · p. 5
  • The composite weights risk, evidence, and factors at 0.4, 0.3, and 0.3, while factor macro-F1 weights all 24 categories equally.Evidence is scored per post, and all 588 posts without marked evidence are at the lowest risk level; every post above that level has evidence.Source: §III · p. 6, Table I · p. 6

IV. Deciding Whether a Change Is Real

The section establishes that author-level resampling can detect paired changes more sensitively than single-split absolute scores, while seed variation and selection effects constrain interpretation. It uses win rates rather than confidence intervals because the resamples overlap.

IV. Deciding Whether a Change Is Real

  • The factor seed range was 0.095 and the risk seed range 0.0183, exceeding nearly all measured effects; paired resampling still resolved gains of 0.2377 and 0.0084.The seed range is a descriptive diagnostic, not a statistical test; the paired instrument could resolve differences outside ±0.014, versus about 0.056 for absolute scores on a single split.Source: §IV · pp. 6–7
  • Across author-disjoint resamples, pooling all 5 folds increased the set from 372 posts by 32 authors to 1,635 posts and reduced the single-factor-score spread from about 0.0215 to roughly 0.0030.Posts by the same person resemble one another, so the effective independent-observation count is nearer 32 than 372; these spreads are not analytic standard errors.Source: §IV · p. 6
  • The protocol accepts paired changes that win on at least 80% of resamples, using win rates because overlapping resamples make confidence intervals around their mean misleading.Three accepted techniques judged by this rule won on 89%, 93%, and 99% of resamples; two others cleared measured effect-size bands.Source: §IV · p. 7
  • Choosing the best of eight candidates can inflate apparent gain by roughly 0.014, treated as an upper bound because resamples overlap and candidates correlate.The bound assumes approximate normality and candidate independence; both assumptions fail to some degree here.Source: §IV · p. 7
  • Single-fold held-out predictions used for cutoff fitting do not match the five-fold averaged predictions submitted at test time, so shared resampling cannot expose that calibration mismatch.The mismatch let a calibration error reach the leaderboard.Source: §IV · p. 8

V. Testing the Playbook

Across 31 pre-specified comparisons, 26 techniques did not survive the task-specific evaluation protocol. The surviving techniques draw on definitions or measured properties of this task, while the results show that many familiar interventions did not yield reliable gains.

V. Testing the Playbook

  • Only 5 of 31 pre-specified techniques survived; 26 had no measurable effect, worsened a component, remained unresolved, or helped only conditionally.Table III presents one row per comparison and reports component-specific scores and protocol-based verdicts.Source: §V · p. 8, Table III · p. 10
  • Scaling from the smallest to mid-sized encoder raised factor score from 0.4495 to 0.6038, but the next size increase was unresolved on factors and risk.The largest model scored 0.6199 on factors, a 0.0161 gain inside the 0.095 seed range; its risk score was 0.7902 versus 0.8001 for the middle size.Source: §V · p. 8
  • Synthetic and pseudo-labeled supervision did not help; generated posts also contained an opening pattern found in 16.8% of generated posts and none of the real posts.Task-adaptive pretraining scored 0.5950 alone versus 0.6132 for the plainest pool member, and adding it to the pool cost 0.0062.Source: §V · pp. 8, 10
  • Class-balanced loss gained +0.0084 macro-F1 on factors and won on 89% of resamples, while the other three tested objective changes did not hold.The other changes included focal loss, asymmetric loss, and difficulty-weighted sampling; their reported differences were unresolved or failed replications.Source: §V · p. 8
  • Two architecturally different members added +0.0157 macro-F1 to a saturated nine-member pool, with gains attributed to less-correlated errors rather than standalone accuracy.The weaker member by standalone score still helped; greedy member search gained 0.0063 on 77% of resamples.Source: §V · pp. 8–9
  • The tested structural modifications did not improve results: the concept channel added no measurable contribution, and the ordinal head scored 0.7363 versus 0.7579 for a flat classifier.The same ordinal head helped on a different architecture.Source: §V · p. 10
  • Threshold fitting on scored posts inflated factor results by 0.0684, while deployment-consistent rate correction gained only +0.0009 on 52% of resamples.The deployment-consistent correction was smaller than the apparent +0.0235 gain obtained when rates were estimated on the whole corpus.Source: §V · p. 11
  • Definition conditioning raised factor macro-F1 from 0.3795 to 0.6172, while the same move scored 0.4830 for evidence against 0.7787 for an ordinary tagger.The model, size, and training data were unchanged; the evidence output lacked a definition to match against.Source: §V · p. 11

VI. Task-Conditioned Technique Selection, and the Fix That Was Not a Technique

The audit’s surviving techniques exploit knowledge or structure specific to the task, while deployment-consistent calibration fixes a separate mismatch between threshold fitting and test-time predictions. The authors use these findings to motivate task-conditioned technique selection: keep a technique only when the task supplies what it needs and controlled measurement supports it.

Task-conditioned technique selection

  • The proposed selection procedure asks what task knowledge a technique needs, whether the task supplies enough of it, and whether a controlled comparison supports keeping it.The audit’s table traces six chains to components and nine to nothing; the authors frame the procedure as a hypothesis for similar future corpora, not a law.Source: §VI · p. 12, Table IV · p. 13
  • Definition conditioning raised the factor score from 0.3795 to 0.6172, but scored 0.4830 on evidence against 0.7787 for an ordinary tagger.The codebook defines the 24 factors but does not define evidence spans.Source: §VI · p. 12
  • Architecturally diverse ensembling gained 0.0157 macro on 93 percent of resamples, while adding same-family members moved 0.6038 and 0.5782 to 0.5755.The added members differed in their errors; the authors do not claim that one member’s weaker standalone score was itself an asset.Source: §VI · p. 12
  • Evidence-clause-scoped symbolic rules gained 0.0018 risk weighted F1 on 99 percent of resamples, whereas applying them to the whole post moved 0.7858 to 0.7841.Scoping raised completed-act-rule precision from 0.284 to 0.667 on triggered posts, but applying the approach to two other rules reduced F1 by 0.068 and 0.066.Source: §VI · p. 12
  • Routing the attempt class to a separate model helped the main model’s weakest class, but routing a non-weak class turned negative.Four techniques were tested with and without the relevant task knowledge; all four changed sign or lost their effect, while class-balanced loss lacked a comparable non-long-tail control.Source: §VI · p. 12

Deployment-consistent calibration

  • +0.0137 factor macro-F1 and +0.0041 composite came from correcting the threshold mismatch, the largest increment to the finished system.Each member’s test scores were mapped to the range of its held-out scores before combination; the correction fit no parameters, used no labels, and required no retraining.Source: §VI · pp. 12–13

VII. The System, and What It Does Not Contain

The submitted system retains components selected through the audit and leaves out techniques that did not survive measurement. Its three outputs use a risk chain, an evidence-tagging panel, and an independent factor pool; the authors also report a factor-pool limitation tied to member selection.

The system and its results

  • TRIDENT retains only departures justified by task knowledge or a measured failure mode; 31 techniques were tested, and only five comparisons produced reliable gains.The finished system departs from plain fine-tuning in 10 places, with six traced to specific task knowledge in Table IV.Source: §VII · p. 14

The risk chain

  • The risk chain scored 0.8203 weighted F1 on the blind leaderboard set and combines risk models before clause-scoped rules and attempt-class routing adjust the level.Its stages include an ensemble of three seeded models, an independently built mental-health encoder, and an entailment model that scores written level descriptions against the post.Source: §VII · pp. 14–15

The evidence panel

  • Seven evidence taggers receive the predicted risk level, and spans survive when at least two proposals match under the task’s span-matching rule.The panel clusters matching spans and returns at most three per post; removing an eighth tagger improved leave-one-out evidence performance by 0.0010, an unresolved result.Source: §VII · pp. 15–16

The factor pool

  • The factor pool scores 0.7045 macro-F1 on the 378-post blind leaderboard set using 11 entailment members, each averaging five fold models.Posts are premises and codebook definitions are hypotheses; the pool uses class-balanced training, diverse architectures, and member score rescaling before averaging in log-odds.Source: §VII · p. 16

The system’s dependencies

  • Risk conditions evidence and marked clauses constrain rules, but the factor pool reads the post independently because risk spans add no factor signal.Training on gold risk spans reached 0.1897, compared with 0.1900 for random text of the same length from the same posts.Source: §VII · p. 16

What the system does not contain

  • Five of the 11 factor members had no individually resolvable contribution, yet the authors kept the larger pool because selecting members on the same held-out predictions risks bias.Proper selection returned +0.0063, compared with +0.0157 for the set fixed in advance.Source: §VII · p. 16

The system and its results

  • The system ranked third of 53 on the public leaderboard, with risk 0.8203, evidence 0.7953, and factor output 0.7045 macro-F1.The standing was at submission close, not the organizers’ later final evaluation; the three component metrics are not on a common scale.Source: §VII · p. 16

VIII. Conclusion

Across 31 comparisons, only five held, supporting task-conditioned technique selection over adding standard techniques by default. The conclusion favors components justified by task knowledge, structure, or measured behavior.

VIII. Conclusion

  • Table V distinguishes six departures grounded in task knowledge from four weaker-grounded choices, including three settled by measurement.Its first departure has no support from the audit, while the final row is the rest of the system.Source: Table V · p. 17
  • 5 of 31 pre-specified comparisons held, 26 did not, and 6 measurably worsened a component.The authors report this audit as the basis for a simpler, task-grounded system.Source: §VIII · p. 17
  • The surviving choices exploit task-supplied definitions and imbalance, or measured ensemble-error properties; other choices were scoped to the task.The authors name this principle task-conditioned technique selection and distinguish it from deployment-consistent calibration.Source: §VIII · p. 17
  • Definition conditioning moved factor score from 0.3795 to 0.6172, while doubling encoder size moved it from 0.6038 to 0.6199, within the seed range.The conclusion also reports that definition conditioning scored 0.4830 against 0.7787 on the evidence output, where the codebook supplies no matching definitions.Source: §VIII · p. 17
  • Fitting cutoffs on the posts being scored inflated results by 0.0684, while correcting the score mismatch was the largest improvement to the finished system.The mismatch arose because threshold fitting used single-model predictions while the submitted system averaged five models within each ensemble member.Source: §VIII · pp. 17–18
  • Before adding a component, the authors recommend identifying the task knowledge, structure, or measured failure mode it uses and verifying that the task supplies enough of it.They present this as a rule for the next corpus of this shape, not a universal claim.Source: §VIII · pp. 17–18

IX. Limitations

The authors identify seven limitations spanning the moving comparison baseline, narrow measurements, unablated settings, multiplicity, and external validity. Together, they bound what the audit can establish beyond this task and sample.

IX. Limitations

  • The incumbent baseline improved during the campaign, so late techniques faced a stronger comparator; only five failed techniques were re-tested against the final system.All five still failed, but the remaining failures were not re-tested.Source: §IX · p. 18
  • The ceiling estimate uses nonrandom near-duplicate posts concentrated among a few authors, so it bounds attainable score rather than locating it.A proper estimate would require re-annotating a random sample with a second clinician set.Source: §IX · p. 18
  • Seed ranges rely on only three factor runs and two risk runs, likely understating variability; no range was established for evidence.Both measured ranges were at operating points below those of the submitted system, leaving near-boundary evidence results unresolved.Source: §IX · p. 18
  • Several techniques were sampled rather than swept, and the large-model versus conditioned-encoder comparison remains unsettled because folds were still improving when training stopped.At three epochs the encoder led by 0.0106, while at six epochs the large model led by 0.0399.Source: §IX · p. 18
  • Every risk model used an author-sequence position marker, but the audit never tested its contribution.The marker was computed within authors and did not cross split boundaries.Source: §IX · p. 18
  • The 31 comparisons create multiplicity risk: thresholds were fixed conventions, no family-wise correction was made, and chance findings may inflate the five positives more than the 26 negatives.The authors designed their protocol to make favorable results harder to obtain, but it does not remove this risk.Source: §IX · p. 18
  • External validity is bounded to one English platform, codebook, and metric, with no prospective workflow evaluation, demographic analysis, or independent replication.The authors claim the pattern only for tasks sharing the described shape; leaderboard ordering may not survive a different author sample.Source: §IX · p. 18

X. Ethical Considerations

The authors describe safeguards for handling sensitive public posts and caution against treating offline scores as evidence of clinical readiness. They also identify surveillance and over-reliance on automated explanations as foreseeable harms.

X. Ethical Considerations

  • The corpus was used only under shared-task terms, was not shared, and all processing ran locally without sending posts to third-party services or hosted models.The authors state that public availability does not remove ethical obligations when content is sensitive and potentially identifiable.Source: §X · pp. 18–19
  • No corpus text is quoted; any needed example would have been author-written to reduce re-identification risk.The authors say no illustrative example proved necessary.Source: §X · p. 19
  • The system is for offline research ranking, not diagnosis or clinical action without a human reading the post; its scores do not establish operational safety or effectiveness.The authors say aggregate scores cannot establish acceptable rates for either missed high-risk posts or false alarms.Source: §X · p. 19
  • Foreseeable harms include surveillance of people who did not seek attention and reviewers stopping their checks because a tool appears explanatory.The authors call for use only within services people have chosen, with an accountable person in the loop.Source: §X · p. 19
  • Clinicians made the judgments by reading distressing material, and the authors say they received its benefit without doing that annotation.Source: §X · p. 19

Related papers

Loading 2609.07766v1…