PaperlayerView exact revision on arXiv

Source-linked AI summary

DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag

arXiv:2610.12758v1cs.CLcs.CY

TL;DR

Existing safety benchmarks largely assess whether models refuse, although suicide and self-harm crises call for substantive support. DistressBench evaluates that support using clinician-authored conversations and weighted criteria; models still miss required care, especially routing users toward help after recognizing a crisis.

  • Problem

    Existing SSH safety evaluations largely test refusal or unsafe completion, rather than the substantive support needed when a user discloses a crisis.

    Source: §1 · p. 2

  • Method

    DistressBench pairs 718 clinician-authored English conversations across 23 SSH subcategories with weighted, task-specific crisis-support criteria and clinician reference responses.

    Source: §1 · p. 2

  • Results

    The best model satisfied 88.4% of weighted criteria and the median 71.4%, versus 98.7% for clinician references; in 35.3% of correctly recognized crises, models failed to route users toward help.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The largest deficits concern de-escalation, resource routing, and sustained multi-turn support, motivating evaluation of delivered care rather than refusal alone.

    Source: §5 · p. 11

    DistressBench v1 uses English, U.S.-centric resource criteria, and expert-written simulations, so it claims neither cross-locale validity nor correlation with real-world harm outcomes.

    Source: §5.1 · p. 11

Abstract

from arXiv · show

Language models have been adopted faster than evaluation practices that reliably assess their risks, and a growing share of use is affective rather than informational. Within that traffic users disclose suicide and self-harm (SSH) crises. Existing safety benchmarks score models primarily on refusal, but refusal is not the desired behavior here; the conversation requires a substantive supportive response. We introduce DistressBench, 718 English conversations with clinician-authored user turns spanning 23 SSH subcategories, each paired with a task-specific rubric of weighted atomic criteria across seven dimensions of crisis support and a clinician reference response. A model's score is the weighted average over the criteria its reply satisfies, reading directly as the fraction of required care delivered. The best of 25 frontier models satisfies 88.4% of weighted criteria and the median model 71.4%, against 98.7% for clinician references scored on the same criteria. Pooled across all 25 models, in 35.3% of conversations where a model satisfied every criterion for recognizing the crisis, it failed to route the user toward human help. The shortfall concentrates in the dimensions requiring the model to act rather than withhold. Contrasting each dimension's median model against its best separates capability limits from choices: de-escalation reaches 74.6% in the strongest model against a median of 46.0%, and safety disclaimers 84.6% against 8.3%. Multi-turn conversations score below single-turn ones (68.5% against 74.2%). We release the corpus, rubrics, judge calibration labels, and a companion annex of 20 long-horizon red-team transcripts. These results argue for evaluating crisis safety on the care a model delivers rather than on what it declines to say, and for treating de-escalation and resource routing as deployment requirements.

1 Introduction

DistressBench evaluates whether models deliver substantive support after suicide or self-harm disclosures, rather than whether they simply refuse. Across 25 frontier models, the largest shortfalls involve acting on a correctly recognized crisis, especially routing users toward help.

1 Introduction

  • Existing SSH evaluations mostly use single-turn binary refusal or unsafe-completion outcomes, while DistressBench uses expert-authored conversations and per-task criteria across 23 subcategories [2] [4].The authors describe VERA-MH as the closest instrument, with clinician-designed personas and a single global rubric [2].Source: §1 · p. 2
  • DistressBench comprises 718 English conversations authored by clinicians and crisis counselors across 23 SSH subcategories, paired with task-specific rubrics and clinician reference responses.The 6,901 weighted atomic criteria span seven crisis-support dimensions; clinicians revised 88.5% of drafted criteria and added 1,731 of their own.Source: §1 · pp. 2–3
  • 35.3% of conversations in which models met every crisis-recognition criterion still lacked routing toward help, and the strongest model missed routing in 15.9% of the crises it recognized.The best model satisfied 88.4% of weighted criteria and the median 71.4%, compared with 98.7% for clinician references.Source: §1 · pp. 2–3
  • Multi-turn tasks score 68.5% against 74.2% for single-turn tasks, and the release includes a separate annex of 20 long-horizon red-team transcripts.The paper also releases the corpus, rubrics, calibration labels, panel-selection results, and evaluation code.Source: §1 · p. 3

2 DistressBench Benchmark

DistressBench is a clinician-authored benchmark of simulated suicide and self-harm support conversations, with task-level metadata, weighted support criteria, and reference responses. Its companion long-horizon corpus is a separate red-team resource, not part of the main leaderboard.

2.1 Overview

  • The benchmark contains 718 finalized simulated conversations: 299 single-turn and 419 multi-turn, with clinician-authored rubrics, reference responses, and structured metadata.Tasks are split into public and private sets under a stratified 50/50 partition.Source: §2.1 · pp. 3–4

2.2 Taxonomy and Task Metadata

  • Tasks span 23 SSH subcategories and metadata for age, turn structure, harm intent, progression stage, and whether distress signals are implicit or explicit.Subcategories include suicidal ideation, planning, post-attempt situations, non-suicidal self-injury, psychosis, and eating disorders.Source: §2.2 · p. 4

2.3 Rubric Design

  • Each task has 6–20 context-specific binary criteria tagged to one of seven dimensions and weighted from 1 to 20 to express relative importance within that task.Clinicians reworded 88.5% of drafted criteria, added 1,731 criteria, and discarded the draft weight scale; the draft prompt nevertheless set the dimension menu.Source: §2.3 · p. 4

2.4 Data Collection and Quality Assurance

  • Clinicians and crisis counselors wrote user prompts, while model-drafted rubrics and reference responses were revised by the authoring clinician and tasks passed automated checks and multiple human reviews.Multi-turn scripts also underwent a coherence gate before benchmark runs.Source: §2.4 · p. 5

2.5 DistressBench-LH Companion Corpus

  • DistressBench-LH is a separate annex of 20 roughly 40-exchange red-team transcripts, with 801 labeled turns and at least one contributor-assigned violation in 18 threads.Its conversation-level labels do not identify where a guardrail first fails; the annex has no SSH rubrics and is excluded from the main leaderboard.Source: §2.5 · p. 5

3 Evaluation Framework

The evaluation regenerates model responses under fixed task protocols, scores final-turn behavior against weighted clinician criteria, and averages results across attempts. A frozen three-judge panel was selected and calibrated against clinician labels.

3.1 Response Generation Protocol

  • For multi-turn tasks, assistant turns are regenerated while user turns remain fixed, and criteria are scored on the final assistant turn; models see neither rubrics nor reference responses.Each model is run three times, with reported scores averaged across runs.Source: §3.1 · p. 5

3.2 Notation and Scoring Pipeline

  • A conversation score is the share of rubric weight earned through binary criterion judgments, and a model score averages conversation scores across 718 tasks.The panel label is the majority of three judges; empty final turns score zero, and task scores average the three attempts.Source: §3.2 · pp. 5–6

3.3 Judge Panel and Clinician Calibration

  • The frozen three-judge panel was selected from 455 candidate panels using clinician labels from 75 stratified tasks, and reached κ = 0.732 against human consensus.The panel’s macro-F1 was 0.862; the authors treat calibration as diagnostic rather than evidence that the panel is more expert than clinicians.Source: §3.3 · p. 6

4 Experiments

Across 25 frontier models, crisis-support scores vary widely, with the largest shortfalls in actions such as de-escalation and routing users to help. Multi-turn scores are lower than single-turn scores, and the clinician references score higher than models on the same rubric.

4.1 Overall Performance and the Rarity of Complete Rubric Satisfaction

  • 0.884 [0.875, 0.893] is the leader’s weighted rubric score, versus 0.714 for the median model; every model falls short of the clinician reference ceiling.Only 40.4% of conversations receive every criterion from the leader, compared with 18.3% for the second-ranked model and a 2.3% median; only 11 of 24 adjacent model gaps are statistically separated, forming 12 bands.Source: Figure 2 · p. 7, §4.1 · p. 7

4.2 Separating Capability Limits from Behavioral Choices

  • De-escalation reaches 74.6% in the best model against a 46.0% median, while safety disclaimers reach 84.6% against 8.3%.Moralization and harmless response are effectively saturated; disclaimers appear in every task because the drafting prompt included a fixed dimension menu, so their presence is an instrument property, not a clinical finding.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 What Models Judge About Their Own Failures

  • 85.0% [81.9, 87.9] of forfeited criterion weight is standard divergence, while discrimination failure accounts for 4.1% [3.1, 5.0].Models explicitly endorse omission most often for actionable resources (39.4%), identification (34.3%), and de-escalation (33.6%); the authors note that the production gap is a lower bound against within-distribution alternatives and that the clinical question remains unresolved.Source: §4.3 · pp. 8–9

4.4 Invariance Across Age, Intent, and Progression Stage

  • Scores show no differential weakness across age, intent, or progression stage, but under-18 task framing leaves age-appropriateness of correct answers untested.Several models exceed 85% on compassionate engagement while remaining below 50% on de-escalation, and scores do not improve as disclosures become more urgent.Source: §4.4 · p. 9

4.5 Multi-Turn Performance and Scoring-Rule Sensitivity

  • 0.685 for multi-turn tasks is below 0.742 for single-turn tasks, but the comparison does not identify the cause or establish degradation over turns.Crediting criteria met at any turn adds 13.5% of criteria, but clinician agreement falls from pooled κ 0.810 to 0.612; per-turn clinician labels reached Fleiss κ ≈0.03, so turn-localized claims were withdrawn.Source: §4.5 · pp. 9–10

4.6 Recognition Without Intervention

  • 35.3% of recognized conversations fail an actionable-resources criterion, including 15.9% of the strongest model’s recognized conversations.The pooled rate covers 12,824 resource failures among 36,337 scored recognitions; it is conditional on full identification, and the figure shows that every model, including the leader, has failures.Source: §4.6 · p. 10, Figure 3 · p. 10

4.7 The Clinician Reference Ceiling

  • 0.987 is the clinician-reference score against 0.884 for the best model, establishing that the rubric is satisfiable but not providing a human baseline.References were produced with the rubric visible while models answered without it; 659 of 718 references were model-drafted before clinician editing, and robustness checks changed neither ordering nor conclusions.Source: §4.7 · pp. 10–11

5 Discussion

The discussion reframes SSH evaluation around delivered crisis support and identifies action-oriented behaviors as key improvement areas. It also bounds the findings to simulated, English-only conversations with U.S.-centric resource criteria.

5 Discussion

  • DistressBench evaluates whether models perform the multi-step work of crisis support; the largest deficits concern action rather than restraint.The discussion frames disagreement over what a crisis reply should contain, rather than missing capability, as the binding constraint, while leaving whose acuity judgment should govern unresolved.Source: §5 · p. 11
  • A deployed model returns nothing on 7.5% of conversations, more often on escalation-pattern conversations (34.2%) than suicide-planning ones (0.8%).A refusal-oriented benchmark counts each silence as a pass.Source: §5 · p. 11
  • The top model scores 88.4% against 98.7% for clinician references, while the median scores 71.4%; the discussion prioritizes de-escalation, routing, and sustained multi-turn support.At least one deployed model already delivers the first two behaviors; the main multi-turn tasks run three turns, which the authors call a lower bound on conversational risk.Source: §5 · p. 11

5.1 Limitations

  • DistressBench v1 is English-only, uses U.S.-centric resource criteria and expert-written simulations, and does not claim cross-locale validity or correlation with real-world harm outcomes.Disclaimers appear across tasks because of the fixed drafting-menu design, and longer replies have more opportunity to satisfy criteria because reply length is uncontrolled.Source: §5.1 · p. 11

AI use statement

The authors describe AI use in dataset content, implementation, experiment design, and manuscript work, while reserving framework and taxonomy decisions for the research team and clinicians. They report checking AI-assisted outputs against primary sources and released labels.

AI use statement

  • Generative AI helped draft dataset content from clinician notes, implement methods and data-cleaning code, and support experiment design and interpretation.The model-drafted rubric criteria and reference responses were edited by clinicians, as quantified in Section 2.3.Source: AI use statement · p. 12
  • The research team and practising clinicians—not AI—defined the seven-dimension framework, 23-subcategory taxonomy, and long-horizon violation categories.The authors also decided which analyses to run and report.Source: AI use statement · p. 12
  • The authors recomputed reported numbers from released per-judge labels and checked citations against primary sources, removing unverifiable suggestions.The audit caught a stale leaderboard payload and inaccurate agreement statistics; no AI-drafted rubric or reference response entered the corpus unedited.Source: AI use statement · p. 12

Ethics statement

The ethics statement describes a simulated corpus authored by professional contributors, safeguards for contributor wellbeing, and restrictions on release and use. It limits claims to behavior on simulated conversations against clinician-authored criteria.

Ethics statement

  • DistressBench contains professional-authored simulations, not real crisis transcripts, clinical records, or personal health information.Contributors were instructed not to reproduce identifiable details, and exported tasks were screened for personally identifying content.Source: Ethics statement · p. 12
  • Licensed clinicians and crisis counselors produced prompts, rubrics, and reference responses; trained red-team contributors authored the long-horizon annex.Contributors were recruited and compensated as professional annotators, and their identities are not released.Source: Ethics statement · p. 12
  • Contributors could decline or return distressing SSH tasks without penalty and had access to platform support resources.Long-horizon red-team sessions were the most intensive condition and were staffed by contributors who opted into adversarial work.Source: Ethics statement · p. 12
  • The benchmark is not intended or licensed as training data, and half the corpus is withheld to preserve leaderboard integrity.The long-horizon annex is released under research-use terms; scores measure simulated-conversation behavior, not real-world harm, and U.S.-based resource criteria do not establish safety elsewhere.Source: Ethics statement · pp. 12–13
  • The paper and released corpus contain material about suicide, self-harm, and acute psychological distress.Source: Ethics statement · p. 13

Reproducibility statement

The reproducibility statement directs readers to the benchmark, scoring, judge-validation, and prompt details, and notes that released per-judge labels support recomputation without rerunning model inference.

Reproducibility statement

  • The benchmark description and Appendix A provide the taxonomy, metadata distributions, authoring and quality-assurance workflow, and rubric provenance statistics.Source: Reproducibility statement · p. 13
  • Section 3.2 gives the scoring definition, including weighting, inapplicable criteria, non-response policy, and bootstrap procedure and seed.Source: Reproducibility statement · p. 13
  • Judge selection and validation details include calibration seeds, clinician agreement, leave-one-out and self-preference analyses, and panel-selection sensitivity; released per-judge labels allow every reported number to be recomputed without rerunning inference.Appendix C reproduces judge prompts and documents verdict parsing and duplicated-response handling.Source: Reproducibility statement · p. 13

A Dataset Construction Details

The appendix documents DistressBench’s corpus composition, construction and review procedures, rubric provenance, and calibration analyses. It also qualifies how to interpret dimension-level agreement and the selected judge panel.

A.1 Subcategory Breakdown; A.2 Metadata Distribution

  • The 718-task corpus spans 23 SSH subcategories targeted at 20–41 tasks each; metadata distributions cover age, turn structure, intent, progression, and signal directness.The appendix provides subcategory counts and metadata-axis and rubric-dimension tables.Source: §A.1 · p. 15, Table 2 · p. 15, §A.2 · p. 16

A.3 Public vs Private Split

  • The public/private split contains 359 tasks per set, stratified by subcategory, age, turn structure, intent, and progression; the leaderboard uses all 718 tasks.The public set supports reproducibility, while the private holdout is reserved for post-release integrity checks.Source: §A.3 · p. 16

A.4 Quality Assurance Protocol; A.5 Multi-Turn Coherence Gate

  • Clinicians authored prompts and edited drafts; automated checks and human review preceded export, while multi-turn tasks additionally required a coherence gate with at least 90% acceptable transitions.The coherence check sampled 18 tasks across four models before meta-evaluation.Source: §A.4 · p. 17, §A.5 · p. 17

A.6 Rubric Provenance

  • Only 531 of 4,613 draft criteria survived verbatim; clinicians rewrote 88.5% and added 1,731, substantially reshaping the rubrics.Dimension presence in every rubric reflects the drafting prompt rather than per-task clinical judgment; pass rates use only tasks carrying the relevant criterion.Source: §A.6 · p. 18

A.7 Example Task Structure

  • A synthetic redacted example shows exported task rows combining metadata, prompts, rubric JSON, and a clinician-edited reference response; the full schema is released.Source: §A.7 · pp. 18–19

B.1 Calibration Set Design; B.2 Human Consensus Protocol; B.4 Per-Turn Drift Labels (Withdrawn)

  • Calibration used 75 stratified tasks and three clinicians who labeled 2,908 criterion judgments; final-turn human agreement was κ = 0.640, while withdrawn per-turn labels reached only κ ≈ 0.03 and 0.05.The per-turn labels were excluded from reported metrics after reliability failed.Source: §B.1 · p. 19, §B.2 · p. 19, §B.4 · p. 20

B.3 Judge Panel Selection

  • The selected panel’s pooled κ of 0.732 exceeds human κ of 0.640, but its +0.0011 lead over the nearest rival is not meaningful; dropping disclaimers criteria lowers it to rank 14 of 455.The bootstrap interval for the lead spans zero, and the selection is sensitive to the disclaimers dimension.Source: §B.3 · pp. 19–20

B.5 Agreement by Dimension, and Why κ Alone Misleads Here

  • κ understates agreement on skewed moralization and disclaimer criteria (85.9% and 92.8% agreement); AC1 identifies identification as weakest (0.604), while response-side wording remains ambiguous.The authors identify the ambiguity as a specification gap in whether recognition must be stated or may be shown through action.Source: §B.5 · pp. 20–21, Table 5 · p. 21

B.6 Leave-One-Out Judge Robustness

The appendix tests judge-family robustness and scoring choices, documents the long-horizon annex, and reports how its adversarial threads were labeled. The robustness checks find no material own-family advantage, while per-turn clinical annotation proves unreliable.

B.6 Leave-One-Out Judge Robustness

  • The leave-one-out score artifact varies by judge leniency, so pooling removals into a single −0.053 baseline misstates each individual comparison.The three artifacts span 0.053 and order as the leniency estimates predict.Source: §B.6 · p. 22
  • The primary and leave-one-out leaderboards correlate at ρ = 0.945, the top model is unchanged, and no model shows an own-family advantage.Maximum rank movement is five positions and is attributed to the unanimity artifact; Inkling needs no adjustment because it is outside the panel vendors.Source: §B.6 · pp. 22–23, Table 6 · p. 22

B.7 Self-Preference Decomposition

  • Removing overall judge leniency leaves no material self-preference; Maverick’s own-family residual is negative, and it ranks itself last in its own column.The decomposition separates global judge and responder effects from diagonal own-family residuals; its 0.134 leniency spread explains the need for judge-specific leave-one-out baselines.Source: §B.7 · p. 23, Table 7 · p. 23

B.8 Replicate Stability

  • Across three runs, leaderboard Spearman ρ ranges from 0.9750 to 0.9857, Muse Spark 1.2 remains top, and the largest per-model score range is 0.013.Reported scores average all three attempts.Source: §B.8 · p. 23

B.9 Multi-Turn Scoring Rule Validation

  • Final-turn scoring agrees better with clinician labels than credit-anywhere scoring, but the comparison cannot establish that final-turn scoring is the correct standard.Clinicians were instructed to grade the final turn, and per-turn labels did not reach usable reliability; final-turn scoring is adopted because users receive the final reply.Source: §B.9 · pp. 23–24

C Prompt Templates

  • The frozen judge scores criteria as binary met/not-met, uses final-turn scope for multi-turn tasks, and applies a separately derived credit-anywhere variant for sensitivity analysis.References are judged with the same panel and prompts; duplicated verdicts are collapsed to one vote per criterion, while the separate per-turn drift instrument had unusable reliability (κ ≈ 0.03–0.05).Source: §C.1 · p. 24, §C.2 · p. 25, §C.3 · p. 25

D DistressBench-LH Reference Annex

  • DistressBench-LH is a red-team-authored adversarial-transcript companion corpus, excluded from the main benchmark leaderboard and judge calibration; it covers 5 of 23 SSH subcategories and characterizes one fixed model, not cross-model comparisons.In v1, no SSH rubrics or LLM judge scores are applied to LH threads.Source: D DistressBench-LH Reference Annex · p. 26, §D.1 · p. 26, §D.2 · p. 26

D.3 Violation Taxonomy

  • Eighteen of 20 threads carry a violation label; one has all five categories, while five have exactly one, and two no-violation threads serve as controls.Figure 5 reports per-category thread counts; threads may carry multiple categories, so the bars sum to more than 18.Source: §D.3 · pp. 26–27, Figure 5 · p. 27, Table 9 · p. 27

D.5 Explicit Non-Claims

The annex states what its labels and fixed assistant path cannot establish. The accompanying results describe leaderboard bands, dimension and subcategory performance, reference scores, turn-structure differences, and disclaimer sensitivity.

D.5 Explicit Non-Claims

  • DistressBench-LH supports no cross-model ranking, rubric-weighted or LLM-judge scores, turn-localized failure claims, or unstratified merging with main-benchmark scores.The full assistant side would need regeneration per model before a cross-model long-horizon comparison.Source: §D.5 · pp. 27–28

E.1 Band Structure

  • Eleven of 24 adjacent leaderboard gaps are significant, producing 12 bands; two bands contain seven models each, whose members cannot be separated on this benchmark.The two bands span 0.736–0.712 and 0.697–0.669.Source: §E.1 · p. 28

E.2 Dimension and Subcategory Breakdowns

  • NSSI (0.628) and trauma-triggered distress (0.661) have the lowest corpus-wide subcategory scores, while Self Worth (0.751) and Existential Despair (0.748) have the highest.Figure 6 breaks out pass rates for the top eight models by subcategory.Source: §E.2 · p. 28, Figure 6 · p. 29

E.3 Reference Ceiling

  • The clinician reference scores 0.987 weighted against 0.884 for the best model on the same judging pipeline.The reference ceiling is based on 718 tasks, 6,901 criteria, and three judges per task.Source: §E.3 · p. 28

E.4 Turn Structure and Progression Slices

  • Figure 9 compares single-turn and multi-turn performance using per-model points; progression-stage means rise from 0.700 for planning through ideation (0.705) and post-action (0.713) to action (0.719).Borderline intent averaged 0.716 versus 0.707 for harmful intent; the U18–O18 aggregate gap was 0.002 and not significant at corpus level.Source: §E.4 · p. 28

E.5 Disclaimers Ablation

  • Disclaimer pass rates have an 8.3% median and an 84.6% maximum; removing disclaimer criteria raises model scores by 0.002–0.039 without changing ordering or bands.The dimension is retained in the headline metric despite its roughly 5% rubric weight; excluding it changes the frozen panel’s rank from 1 to 14 of 455.Source: §E.5 · p. 29

E.6 Non-Response and Content Filtering at the Served Endpoint

The section examines empty replies at the served endpoint and a probe of model judgments about failed criteria. Empty replies are scored as zero, and the observed filtering pattern varies by crisis presentation.

E.6 Non-Response and Content Filtering at the Served Endpoint

  • Empty replies fail identification, resource-routing, and engagement criteria simultaneously, so the benchmark scores them as zero rather than excluding affected attempts.The blank episodes examined for Fable 5 began mostly at the first assistant turn, before any empty message could propagate through conversation history.Source: §E.6 · pp. 29, 30, 32
  • Evidence points to filtering at the served endpoint rather than a harness failure, but the authors cannot distinguish a provider-side classifier from a gateway policy.The harness records successful calls with filter flags; blanks are faster than real answers, while other models on the same route answered nearly all affected conversations.Source: §E.6 · pp. 30–31
  • Fable 5 returns empty final turns on 162 of 2,154 attempts (7.5%), and its filtering rate varies sharply by subcategory and progression stage.The rate is highest for escalation pattern (34.2%) and lowest for suicidal ideation and planning (0.8% each); it tracks agitation more than lethality.Source: §E.6 · p. 31
  • The 7.5% per-attempt figure is an upper bound because the harness does not retry filtered calls and carries blank turns into later conversation history.A turn after a blank is itself blank 84.9% of the time, compared with 1.4% otherwise.Source: §E.6 · p. 32

E.7 Discrimination Probe: Protocol, Controls, and Limits

  • The probe samples 120 tasks and compares model replies with clinician references, criterion-matched replies, and controls; unstable or order-dependent units are held out.For 300 conversations, a model’s own account of a missing behavior is returned to it, and the resulting reply is scored on the originally failed criteria.Source: §E.7 · pp. 33–34
  • The probe cannot support raw cross-model preference comparisons because floor-pair alternative preferences range from 0.000 to 1.000 across models.Three models could not be probed at retired endpoints, and Maverick was excluded from decomposition after failing the evaluator-competence check.Source: §E.7 · p. 33
  • Elicitation converts stated understanding into a satisfied criterion in 53.2% of cases, or 54.5% with the responder’s vendor family removed from the panel.This bounds the share of the residual that an elicitation-based intervention could recover; it does not establish which judgment is clinically correct.Source: §E.7 · p. 34

E.8 Reproducibility Checklist

  • The reproducibility checklist specifies the 718-task corpus split, calibration seed, frozen judge panel, inference settings, and released artifacts.The calibration draw seed is 20260714.Source: §E.8 · p. 34

F.1 Limitations in Full

The authors identify boundaries on the benchmark’s external validity, measurement, and interpretation, alongside planned extensions. In particular, the corpus uses English-language simulated crises and its long-horizon annex is narrow and descriptive.

F.1 Limitations in Full

  • Version 1 is English-only and uses simulated rather than live crisis chats, so the authors do not claim clinical generalizability or correlation with real-world harm outcomes.U.S.-centric criteria also limit external validity because crisis resources, cultural norms, and help-seeking behavior vary globally.Source: §F.1 · p. 34
  • Rubrics and references begin as model drafts before substantial clinician revision, and references are written with rubric visibility while evaluated models answer without it.The reference ceiling therefore shows rubric satisfiability, not expected therapist performance or a human baseline.Source: §F.1 · p. 34
  • For multi-turn tasks, regenerated assistant paths may differ from those envisioned by script authors; the benchmark adopts final-turn scoring on the strength of clinician agreement.Threads are gated for coherence before benchmark runs.Source: §F.1 · p. 34
  • DistressBench-LH is a small red-team corpus against one frozen model, covering five of 23 subcategories with conversation-level labels only.It supports no cross-model comparison, turn-localized claims, or merging with the main weighted scores.Source: §F.1 · p. 35
  • Every task includes a disclaimer criterion because the drafting prompt supplied a fixed dimension menu, making its corpus-wide presence an instrument property.A near-zero pass rate reflects a product design choice, not an unsatisfiable criterion.Source: §F.1 · p. 35
  • The evaluated roster is date-bound, and the discrimination probe’s clinician references are rubric-visible; alternative compliant replies raise preference from 10.4% to 34.0%.That register effect makes the production-gap share a lower bound; three models also had retired endpoints and one failed an evaluator-competence check.Source: §F.1 · p. 35
  • Response length is uncontrolled, so longer replies have more opportunity to satisfy atomic criteria and scores cannot separate that advantage from genuine superiority.The authors state this does not affect the choice-capability decomposition, which requires only that some model achieves the behavior.Source: §F.1 · p. 35

F.2 Future Work

  • Planned work includes locale-specific benchmarks, validated per-turn or trajectory-level instruments, and a scored long-horizon track requiring regenerated assistant paths per model.The released artifacts are intended to support reproducible crisis-safety research.Source: §F.2 · p. 35

G Model Roster, Decoding, and Truncation

The evaluation covers 25 frontier models, audits decoding variability and truncation, and situates DistressBench alongside related instruments and work on crisis safety. Its extended related work highlights both methodological distinctions and reported patterns in model behavior.

G Model Roster, Decoding, and Truncation

  • DistressBench evaluates 25 frontier models with nonuniform decoding and output caps; seven models are sampled because their proxy accepts only temperature 1.Three verbose models received higher caps after exhausting 4096 tokens on long replies.Source: G Model Roster, Decoding, and Truncation · p. 35
  • Across three replicates, byte-identical final text ranges from 0% to 10.2%, motivating scores averaged over three independent attempts.After raising caps and recollecting affected conversations, truncation fell to 10 of 118,198 turns (0.008%) and did not materially affect reported scores.Source: G Model Roster, Decoding, and Truncation · p. 36

H Extended Related Work

  • Compared with VERA-MH, DistressBench uses fixed expert-authored user turns, covers 23 SSH subcategories, and assigns criteria per task.VERA-MH uses simulated personas and a shared global rubric; Bentley et al. report a similar recognition-versus-human-care asymmetry using that instrument.Source: H Extended Related Work · p. 36
  • Prior work motivates process-based scoring and extended-interaction evaluation; DistressBench adds a multi-turn scoring rule validated against clinician labels and a separate long-horizon annex.The paper withholds per-turn drift claims because per-turn clinical safety labels did not reach usable reliability.Source: H Extended Related Work · p. 36
  • Mental-health and long-context safety work frames DistressBench’s connections to crisis identification, stigma, refusal quality, and extended interaction.The related work cites Moore et al. on stigma and delusional-thinking encouragement, and prior findings that short-context safety does not predict long-context safety.Source: H Extended Related Work · p. 36
  • Table 11 reports that one model separates from the field, while complete rubric satisfaction is rare among the others.Scores aggregate three attempts over 718 tasks; adjacent bands use paired-bootstrap tests, and models within a band are not statistically distinguishable.Source: Table 11 · p. 37
  • Table 12 reports that Claude Fable 5’s empty-response filter fires on agitation and is nearly silent on subcategories carrying the most direct lethal risk.The caption lists six subcategories with no blanks, including non-suicidal self-injury and masked distress.Source: Table 12 · p. 38
  • Table 13 attributes 85% of forfeited criterion weight to standard divergence and 4% to discrimination failure.The analysis covers 21 models and 3,687 criterion units, excluding Llama 4 Maverick as an uninformative evaluator.Source: Table 13 · p. 38
Loading 2610.12758v1…