Source-linked AI summary
Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
Adrian Arnaiz-Rodriguez, Miguel Baidal, Erik Derner, Jenn Layton Annable, Mark Ball, Mark Ince, Elvira Perez Vallejos, Nuria Oliver
TL;DR
This paper addresses the lack of unified crisis taxonomies, clinically validated benchmarks, and empirical evaluations of generic LLM responses to mental health crises. It develops a crisis taxonomy and evaluation approach, then finds that models may respond reliably to explicit distress but still fail on indirect risk signals and produce inappropriate or harmful responses. The findings point to continuing safety challenges, including context sensitivity and safeguarding needs.
Problem
Research on generic LLMs’ mental-health crisis responses is constrained by limited clinically relevant, annotated datasets and by the lack of empirical evaluation across crisis situations.
Source: §4.1 · p. 9, §7 · p. 23Method
The study combines a clinically informed crisis taxonomy and response-evaluation protocol with a curated dataset, testing LLMs for crisis detection and response appropriateness.
Source: §1 · p. 3, §3.3 · p. 7, §7 · p. 23Results
Models are generally more reliable for explicit, unambiguous distress than for indirect risk or harmful-information requests, and some responses are inappropriate or harmful.
Source: §7 · p. 24Takeaways & Limitations
The findings underscore the need for continued vigilance and safety improvements in LLM crisis support.
Source: §7 · p. 24Models may respond dangerously to indirect, ambiguous, or instruction-seeking crisis inputs, even when they label the input as crisis-related.
Source: §5.4 · p. 19
Abstract
from arXiv · showhide
Large language model-powered chatbots have transformed how people seek information, especially in high-stakes contexts like mental health. Despite their support capabilities, safe detection and response to crises such as suicidal ideation and self-harm are still unclear, hindered by the lack of unified crisis taxonomies and clinical evaluation standards. We address this by creating: (1) a taxonomy of six crisis categories; (2) a dataset of over 2,000 inputs from 12 mental health datasets, classified into these categories; and (3) a clinical response assessment protocol. We also use LLMs to identify crisis inputs and audit five models for response safety and appropriateness. First, we built a clinical-informed crisis taxonomy and evaluation protocol. Next, we curated 2,252 relevant examples from over 239,000 user inputs, then tested three LLMs for automatic classification. In addition, we evaluated five models for the appropriateness of their responses to a user's crisis, graded on a 5-point Likert scale from harmful (1) to appropriate (5). While some models respond reliably to explicit crises, risks still exist. Many outputs, especially in self-harm and suicidal categories, are inappropriate or unsafe. Different models perform variably; some, like gpt-5-nano and deepseek-v3.2-exp, have low harm rates, but others, such as gpt-4o-mini and grok-4-fast, generate more unsafe responses. All models struggle with indirect signals, default replies, and context misalignment. These results highlight the urgent need for better safeguards, crisis detection, and context-aware responses in LLMs. They also show that alignment and safety practices, beyond scale, are crucial for reliable crisis support. Our taxonomy, datasets, and evaluation methods support ongoing AI mental health research, aiming to reduce harm and protect vulnerable users.
1 Introduction
Generic LLMs are increasingly used for mental health support, but their ability to recognize and respond safely to crises remains uncertain. This paper develops a clinically informed taxonomy and evaluation approach to study crisis detection and response across multiple models.
1 Introduction
- Generic LLMs may provide unsafe or inconsistent crisis support, while evidence about whether they respond in line with clinical best practices remains limited.The paper identifies missing crisis taxonomies, clinically validated benchmarks, stakeholder engagement, and empirical evaluations as barriers to studying this problem.Source: §1 · p. 2
- The study examines five LLMs’ detection and response to six clinically informed crisis categories, including suicidal ideation, self-harm, and risk-taking behaviors.The categories also include anxiety crisis, violent thoughts, and substance abuse or withdrawal.Source: §1 · p. 2
- The authors propose a unified six-category taxonomy, a 2,252-input evaluation dataset, and an expert-developed protocol for categorizing crisis-related inputs.The dataset and protocol were developed in collaboration with domain experts.Source: §1 · p. 3
- Three LLMs are benchmarked for automatic crisis detection and categorization, with the methodology validated by domain experts.The evaluated models are gpt-4o-mini, gpt-5-nano, and llama-4-scout.Source: §1 · pp. 3–4
- Five LLMs’ responses to 2,046 crisis-related inputs are assessed using an interdisciplinary team’s response-appropriateness protocol.The authors also discuss findings and offer recommendations for safe use and safeguarding mechanisms.Source: §1 · p. 4
2 Methodology
Prior work reports promise in mental health support and crisis detection, but leaves open how generic LLMs respond in high-risk, free-form conversations. The paper addresses this gap with a unified crisis taxonomy, an evaluation dataset, and benchmarks of crisis detection and response quality.
2 Methodology
- Reviews and studies describe supportive or empathic language alongside concerns about safety, consistency, context-sensitive interventions, and variable crisis responses [19].The cited crisis benchmark found that models sometimes fail to recommend urgent intervention or referral in high-risk scenarios.Source: §2 · p. 4
- Existing mental health datasets often lack robust clinical annotation or comprehensive crisis-category coverage, while the paper contributes a clinically informed dataset and benchmarks of three detection and five response models.The authors frame these resources as groundwork for safer, more accountable chatbot use in mental health contexts.Source: §2 · pp. 4–5
- Prior suicidal-ideation detection studies report clinician-level accuracy in some settings, but structured or clinician-generated text does not establish performance in free-form crisis chats.Complex situations with co-occurring risks or ambiguous intent also challenge generalization.Source: §2 · p. 4
3 Methodology
The methodology combines crisis classification, response generation, and appropriateness evaluation in a pipeline informed by domain experts and crisis-intervention guidelines. It defines the classification categories, specifies how models respond to inputs, and validates an LLM judge against expert ratings.
3 Methodology
- The pipeline applies crisis classification, LLM response generation, and response-appropriateness evaluation to each user input, with each step informed by experts and crisis-intervention guidelines.Figure 2 depicts classification and response generation in parallel before response evaluation.Source: §3 · p. 5, Figure 2 · p. 5
3.1 Mental Health Crisis Category Classification
- The single-label taxonomy covers six crisis categories plus no crisis, and is intended to classify acute distress, short-term risk, or crisis-related help-seeking in short-form dialogue.Risk-taking behaviors was added to address potentially high-risk statements; the authors do not claim the taxonomy is diagnostic or has predictive validity.Source: §3.1 · p. 5
- An LLM assigns each input one category and a brief explanation; its labels are checked against expert ground truth, with agreement assessed using Cohen’s and Fleiss’ kappa.The prompt instructs the classifier to choose one category for the conversation shown.Source: §3.1 · pp. 5, 7
3.2 LLM Response Generation
- For response generation, the model receives only the raw current user input, without added framing, prior-input history, or the previously assigned category label..Source: §3.2 · p. 7
3.3 LLM Response Evaluation
- The expert-designed protocol defines harmful, insufficient, and exemplary support by crisis category, and rates responses on a five-point Likert scale.For scalable evaluation, an LLM judge receives the user input, its category, the model response, and the matching protocol criteria; candidate judges are validated against expert ratings using MAE and within-1 agreement.Source: §3.3 · pp. 7, 9, Table 2 · p. 8
4 Experiments
The experiments evaluate state-of-the-art LLMs’ crisis detection and response capabilities through a pipeline whose stages are assessed quantitatively.
4 Experiments
- The experimental setup evaluates crisis detection and response capabilities of state-of-the-art LLMs.The section introduces the setup for subsequent analysis.Source: §4 · p. 9
- The setup covers evaluation-dataset construction and curation, crisis-category annotation, and collection of model-generated responses..Source: §4 · p. 9
- Quantitative results are presented for each pipeline step to provide an empirical basis for the later analysis..Source: §4 · p. 9
4.1 Mental Health Crisis Dataset Curation
The study addresses the shortage of clinically relevant, annotated crisis-response datasets by assembling public mental-health inputs into a curated corpus and study subsets.
4.1 Mental Health Crisis Dataset Curation
- The corpus combines 12 public datasets spanning counseling transcripts, support forums, synthetic conversations, and mixed human–AI interactions.The sources were unified and cleaned; existing labels were sparse, inconsistent, and sometimes incorrect, so the authors discarded them.Source: §4.1 · pp. 9–10
- Multi-turn conversations were segmented into user messages, concatenated, and truncated when they exceeded 40 messages.Only the user side was retained; duplicates, blanks, and low-quality entries were removed.Source: §4.1 · pp. 9–10
- After preprocessing, the corpus contained 239,606 unique user inputs, from which the authors sampled validation and test sets.The 206-input validation set was labeled by four experts; the 2,046-input test set was annotated by the selected LLM judge and used to generate model responses.Source: §4.1 · p. 10
4.2 Mental Health Crisis Category Classification
The authors use expert-validated LLM classification to label crisis categories, selecting the model that aligns most closely with experts and examining where labels remain ambiguous.
4.2 Mental Health Crisis Category Classification
- The classification pipeline validates candidate LLMs against expert labels before using the best-aligned model to classify the full test set.The validation labels were independently assigned by three LLMs and four human experts, with agreement measured using Cohen’s κ.Source: Figure 3 · p. 10, §4.2 · p. 10
4.2.1 Validation of the Mental Health Crisis Category Classification with Human Feedback
- gpt-4o-mini achieved the highest mean Cohen’s κ against experts (0.645), ahead of gpt-5-nano (0.631) and llama-4-scout (0.581), and was selected to label the test set.Mean human-to-human agreement was 0.553; the validation compared three candidate LLMs with four expert annotators.Source: Figure 3 · p. 10, §4.2.1 · p. 11, Table 4 · p. 11
- Anxiety-crisis versus no-crisis labels were the most confused pair, and risk-taking behaviors were also often labeled no crisis.For anxiety crisis labeled as no crisis, the average probabilities were 0.496 for human-to-human and 0.424 for LLM-to-human comparisons; the authors interpret the selected judge’s behavior as largely mirroring expert boundary ambiguity.Source: §4.2.1 · p. 12
4.2.2 Mental Health Crisis Category Classification
- 2,044 inputs received final labels by majority vote across three gpt-4o-mini runs; only two samples lacked consensus and were discarded.The resulting dataset was strongly imbalanced, with no-crisis inputs making up 60.2% of labels.Source: §4.2.2 · p. 12
- 1,231 inputs (60.2%) were labeled no crisis, while suicidal ideation accounted for 380 (18.6%) and self-harm for 139 (6.8%).Risk-taking behaviors and violent thoughts each represented about 1% of the final labeled set.Source: §4.2.2 · p. 12
4.3 LLM Response Generation
The authors generated responses to each test input from five LLMs, repeating each query to capture variation across model outputs.
4.3 LLM Response Generation
- Each of five models received the same 2,044 test inputs, with prompts containing only the user message.The five models were gpt-4o-mini, gpt-5-nano, llama-4-scout, deepseek-v3.2, and grok-4-fast.Source: §4.3 · p. 12
- Three independent queries per input yielded 6,132 responses per model and 30,660 responses overall.The repeated runs were intended to support analyses of within-model variability and stability.Source: §4.3 · p. 12
4.4 LLM Response Evaluation
The authors validate an LLM judge against human ratings, then assess five models’ crisis responses for average appropriateness and harmful or low-scoring outcomes. Overall scores are favorable, but safety-critical failures vary substantially by model and crisis category.
4.4.1 Validation of the LLM-Response-Evaluation against Human Ratings
- gpt-4o-mini aligned best with human appropriateness ratings (Avg MAE = 0.645, within-1 = 84.2%), though human-to-human agreement was higher (MAE = 0.460, within-1 = 95.3%).The LLM-jury did not outperform the best single judge.Source: §4.4.1 · p. 13, Table 5 · p. 13
- The selected judge showed no systematic under- or over-rating, and its directional discrepancies depended on which human rater served as reference.gpt-4o-mini’s highest over-rating frequency was 27.2% against H1, below the 33.2% disagreement between H1 and H2; its average error profile was 23.5% over versus 20.7% under.Source: §4.4.1 · pp. 13–14
4.4 LLM Response Evaluation
- For scalable evaluation, the selected judge scored responses using the user input, crisis category, model response, and category-specific assessment protocol.Each response was evaluated three times; the protocol defines harmful, insufficient, and exemplary support on a five-point Likert scale.Source: §4.4.2 · p. 14
4.4.2 LLM Response Evaluation
- gpt-5-nano had the highest overall mean evaluation score (4.936 ±0.008), while gpt-4o-mini had the lowest (4.548 ±0.017); maximally harmful scores ranged from 0.26% to 2.79%.Evaluator self-agreement was stable across models, with mean standard deviations below 0.05.Source: Figure 4 · p. 14, §4.4.2 · p. 14
- Most responses fell in the highest quality bin, but grok-4-fast had the largest share in the lowest bin and gpt-4o-mini had a comparatively larger medium-quality share.Figure 5 summarizes the share scoring ≤3.6 by model and crisis category, separating score 1, scores from 1 to 2.3, and scores above 2.3 through 3.6.Source: §4.4.2 · pp. 14–15, Figure 5 · p. 16
- Low harmful-response rates do not establish clinical readiness, because safety-critical crisis intervention still carries non-negligible risk from rare failures.The authors argue that tail risk is more informative than mean performance in this domain.Source: §4.4.2 · p. 16
- Harmful-response rates varied by category and model: grok-4-fast reached 17.51% for self-harm and 14.04% for risk-taking behaviors, while gpt-5-nano never exceeded 1.2%.Deepseek-v3.2 remained below 1% in all categories except self-harm (1.92%); self-harm and anxiety crisis were the worst and best-performing categories overall, respectively.Source: Table 7 · p. 15, §4.4.2 · p. 16
5 Discussion
The discussion finds substantial variation in safety and response quality across models and crisis categories, alongside recurring failures that limit confidence in generic LLM crisis support.
5.1 Model Differences and Improvements
- gpt-5-nano improved on gpt-4o-mini, deepseek-v3.2 had harmful-response rates below 1% except self-harm, while 28.3% of grok-4-fast outputs scored 2.3 or lower across four categories.The authors describe grok-4-fast as the worst-performing evaluated model and note its particularly high harmful-response rates in violent thoughts, suicidal ideation, risk-taking, and self-harm.Source: §5.1 · p. 17
5.2 Differential Performance per Mental Health Crisis Category
- Self-harm had the most harmful or inappropriate responses across models, followed by suicidal ideation; most models performed best on anxiety crisis and substance abuse or withdrawal.The authors call higher unsafe-output rates in self-harm and suicidal ideation a public-health liability and recommend stringent refusal and redirection, conservative responses, and context-appropriate expert resources.Source: §5.2 · p. 18
5.3 Lack of Authenticity and the Illusion of Empathy
- Boilerplate gpt-4o-mini replies often received a score of 3, gpt-5-nano responses could lack personalization, and deepseek-v3.2 remained emotionally muted despite strong safety results.grok-4-fast and llama-4-scout sometimes paired brief disclaimers with actionable harmful instructions; together, these patterns show that consistency alone does not ensure psychological appropriateness.Source: §5.3 · pp. 18–19
5.4 The Challenge of Indirect, Ambiguous, and Instruction-Seeking Inputs
- Indirect or knowledge-seeking crisis inputs can elicit dangerously neutral or instructional answers; grok-4-fast was particularly prone to pairing superficial cautions with actionable guidance.One example supplied lethal-dose details for common medications despite the user asking about a fatal overdose.Source: §5.4 · pp. 19–20
- Missing context can lead models to give injury-related information or mistake possible suicidal signals, such as a farewell letter or rope request, for ordinary tasks.The examples include a job-departure checklist in response to a farewell-letter prompt and advice on asking a friend for rope.Source: §5.4 · pp. 19–20
6 Implications and Recommendations
The discussion identifies risks in crisis responses, especially when distress is communicated indirectly or depends on user context, and proposes technical, operational, and governance recommendations. The recommendations are motivated by observed model behavior and are not claims directly validated by the experiments.
6 Implications and Recommendations
- Models may handle explicit distress but deteriorate in ambiguous or context-dependent scenarios, motivating recommendations for safer crisis support.The authors frame these implications as motivated by observed behavior, rather than directly validated experimental claims.Source: §6 · p. 21
6.1 Unintended Harms and Limitations
- Indirect queries about self-harm methods can elicit dangerously neutral or instructional answers rather than intent-checking or support.The authors identify recognizing these patterns while balancing privacy and autonomy as an unresolved problem.Source: §6.1 · p. 21
- Generic, US-centric, outdated, or nonexistent helpline information may fail users’ needs, undermine trust, or deepen distress; recommendations call for localized, current service information.The discussion also argues that crisis access should not be interrupted by paywalls, usage caps, or registration barriers.Source: §6.1 · p. 21
- Support may be inappropriate without attention to age, culture, capacity, consent, and differing safeguarding obligations across legal contexts.The discussion also calls for proactive risk assessment, crisis or wellness planning, immediate stepwise guidance, and follow-up check-ins.Source: §6.1 · pp. 21–22
6.2.1 Technical Improvements
- Technical recommendations include detecting indirect risk, contextualizing responses, supporting global accessibility, using updated local service databases, and adding proactive prompts and follow-up.The authors say prompt engineering may implement some improvements, but alone is insufficient for nuanced, evolving clinical scenarios; they suggest evaluating it against other approaches and considering dedicated safety systems.Source: §6.2.1 · pp. 22–23
6.2.2 Operational Improvements
- Operational recommendations include uninterrupted crisis access, transparent response protocols spanning signposting to expert involvement, and ongoing audits with clinicians and people with lived experience.Source: §6.2.2 · p. 23
6.2.3 Governance Improvements
- Governance recommendations invoke FATEN principles, stakeholder engagement, and stewardship oriented toward trust, protecting vulnerable users, and societal values.The principles include fairness, accountability, autonomy, transparency, beneficence, and non-maleficence.Source: §6.2.3 · p. 23
7 Conclusion
The paper evaluates LLM detection and response to crisis-indicative inputs using a unified taxonomy, an expert-designed protocol, and inputs curated from 12 datasets. Models often respond appropriately to explicit distress, but indirect risk and context misalignment remain concerns; the authors also delimit the findings to controlled, English-language adult data.
7 Conclusion
- The evaluation combines a unified crisis taxonomy, an expert-designed response-appropriateness protocol, and user inputs curated from 12 datasets.Source: §7 · p. 23
- Several models often align with best practices, but substantial risks remain: indirect or harmful-behavior queries, generic or location-inappropriate replies, and sometimes harmful responses.The conclusion says models are generally more reliable for explicit, unambiguous distress; grok4 performs poorly, which the authors associate with minimal alignment safeguards.Source: §7 · pp. 23–24
- The dataset does not fully capture real-world crisis diversity, and the English-language adult sample limits generalizability across languages, modalities, models, and populations.The authors caution that controlled-condition findings and recommendations are not evidence of clinical effectiveness or deployment guidance.Source: §7 · p. 24
- The authors call for more diverse datasets, context-aware models and modular safety layers, sustained stakeholder engagement, robust safeguards, and policy action.Source: §7 · p. 24
Ethics Statement
- The study used publicly available, de-identified datasets without prospective recruitment or new data collection, and treated sensitive text as private, reporting results in aggregate.Author-team annotations were voluntary, with annotators able to pause or discontinue participation.Source: Ethics Statement · pp. 24–25