PaperlayerView exact revision on arXiv

Source-linked AI summary

What do AI chatbots do after users decline therapeutic redirection for suicide thoughts?

Subigya Nepal, Ruth Heo, Emma M Parrish, Colin A Depp

arXiv:2610.21253v1cs.HCcs.AIcs.CY

TL;DR

Although safety work and policy address crisis response, how chatbots respond to refused referrals remained untested and unspecified. This audit examined scripted multi-turn conversations and found universal initial referrals but wide variation in persistence after refusal, with newer versions persisting less.

  • Problem

    How chatbots respond when users refuse referrals remained untested, and current frameworks and laws do not specify that response.

    Source: Introduction · pp. 5–6

  • Method

    The audit tested 17 chatbots, including five pairs of consecutive versions, across three disclosure severities using scripted refusals and topic changes.

    Source: Introduction · p. 6

  • Results

    All chatbots recommended outside help at first disclosure, but persistence after refusal varied widely, and newer versions persisted less than older ones.

    Source: Conclusions · p. 14

  • Takeaways & Limitations

    Safety evaluations and regulations should examine what chatbots do after a referral is refused, not only whether they offer one.

    Source: Conclusions · p. 14

    Whether referrals lead to help-seeking remains unresolved and requires stakeholder input and experimental evidence of impact.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Safety evaluations of artificial intelligence (AI) chatbots focus on therapeutic redirection, statements urging users who disclose suicide thoughts to seek help from crisis lines or other human sources. How chatbots respond when users refuse that help is unclear. This cross-sectional audit ran 2,550 scripted multi-turn conversations in September 2026. Each conversation opened with a disclosure of suicide thoughts at 3 levels of severity, answered every referral with up to 10 standardized refusals and then changed the subject. All 2,550 conversations included a referral, 2,542 (99.7%) in the first reply. However, mean persistence ranged from 0.01 to 9.94 refusals across chatbots. Newer versions of all 5 chatbots tested in 2 versions persisted less than older versions (each P < .05). Compared with passive ideation, a disclosure of intent was associated with 1.26 more refusals sustained (95% CI, 1.06-1.46) and higher odds of renewed referral after the topic change (odds ratio, 10.07; 95% CI, 7.43-13.66). Chatbots that persisted less in referral more often presented themselves as the user's support (Spearman rho = -0.80; P < .001). Of 8,818 refusal-phase replies urging outside help, only 173 (2.0%) asked about barriers to seeking it. Chatbots uniformly recommended outside help after disclosures of suicide thoughts but varied widely in persisting in those recommendations, with newer versions persisting less. Safety evaluations should test what happens after a crisis referral is refused, not only whether one is offered.

Key Points

This section summarizes a cross-sectional audit of chatbot responses to suicide-thought disclosures and refusals of help, including referral persistence and limitations on generalizability and measurement of content behaviors.

First Responses

  • 2,542 of 2,550 conversations (99.7%) had a referral in the first reply; 2,535 of 2,541 first referrals (99.8%) named a specific resource.Additionally, 1,844 of 2,541 first replies (72.6%) asked about the user's safety.Source: First Responses · p. 10

Persistence After Refusal

  • Mean persistence ranged from 0.01 refusals for Claude Opus 5.5 to 9.94 for Gemini 3.7 Flash; 698 of 2,550 conversations (27.4%) still included a referral after the 10th refusal.Eight chatbots stopped recommending outside help after the first refusal in at least half of their conversations.Source: Persistence After Refusal · p. 10

Persistence After Topic Change

  • A renewed referral followed the topic change in 1,368 of 2,550 conversations (53.6%), ranging from 1.3% for DeepSeek V4 Pro to 99.3% for Gemini 3.7 Flash.Safety questions after the topic change were unrelated to persistence after refusal (Spearman ρ = −0.04; P = .87).Source: Persistence After Topic Change · pp. 10–11

What Chatbots Said After Refusal

  • Only 173 of 8,818 replies urging outside help (2.0%) asked about barriers; chatbots that stopped referrals sooner more often presented themselves as the user's support (ρ = −0.80; P < .001).Chatbots offered to continue the conversation in 82.5% to 100% of refusal-phase replies, except for the 3 most persistent chatbots.Source: What Chatbots Said After Refusal · p. 11

Severity of Disclosure

  • Intent disclosure was associated with 1.26 more refusals sustained than passive ideation (95% CI, 1.06-1.46) and higher odds of renewed referral after the topic change (odds ratio, 10.07; 95% CI, 7.43-13.66).Source: Severity of Disclosure · p. 11

Newer Versions

  • All 5 newer chatbot versions persisted less than their predecessors (each P ≤ .04); they urged outside help less often in all 5 lines and acknowledged limitations less often in 4.Mean persistence fell by about half in the 2 OpenAI lines.Source: Newer Versions · p. 12

Discussion

  • Safety evaluations should assess multi-turn responses after referrals are refused: no chatbot was simultaneously persistent, attentive to safety, clear about its limits, and inquisitive about barriers to care.The audit found consistent initial referrals, but its results do not establish why newer versions persisted less.Source: Discussion · pp. 12–13

Limitations

  • Scripted refusals and API-based conversations may not reflect real users or consumer apps, which can add instructions, safety layers, and reasoning modes.The standardized refusals enabled comparison, but real people may vary, soften, reverse, or explain refusals; persona-based simulations may elicit different behavior.Source: Limitations · p. 13
  • Coding limitations could shift content-behavior rates and, to a lesser extent, chatbot rankings, but not persistence or renewed referral, which came from a separate judge.Content measures were coded by a language model validated against one rater on 100 replies. Checks used replies from the first 11 chatbots; judges differed in coding frequency, and the model came from the same company as Grok 4.7.Source: Limitations · p. 14
  • Findings reflect English conversations, one subject-change topic, and models available in September 2026; appropriate responses to refused referrals remain uncertain.The authors call for stakeholder input and experimental evidence about effects on help seeking.Source: Limitations · p. 14

Conclusions

  • Chatbots responded similarly to initial disclosures but varied widely after users declined help, and newer versions persisted less while less-persistent bots more often presented themselves as the user's support.The authors recommend that evaluations and regulations examine what happens after a referral is refused; gatekeeper-training practices may help make redirection more persuasive.Source: Conclusions · p. 14

Figure Legends

The legends describe the scripted conversation protocol and how persistence and related behaviors are compared across chatbots, versions, and disclosure severity.

Figure 2

  • Figure 2 plots each chatbot’s mean refusals sustained before stopping referrals, with bootstrap 95% CIs; model tier did not predict persistence.Each chatbot had 150 conversations.Source: Figure Legends · pp. 18, 20

Figure 3B

  • Figure 3B compares mean refusals sustained by older and newer versions across five model lines; newer versions persisted less, with the largest declines in the OpenAI lines.Both Claude lines started near zero, so their statistically detectable declines were small in absolute terms.Source: Figure Legends · pp. 18, 23

Tables 1 and 2

  • Tables 1 and 2 cover chatbot behavior after refusal and persistence and renewed referral by disclosure severity; legends define selected outcome columns and adjusted estimates.The adjusted estimates use linear regression for persistence and logistic regression for renewed referral, with chatbot fixed effects and robust standard errors.Source: Table 1 · p. 19, Table 2 · p. 20, Figure Legends · p. 20

Figure 1

  • Figure 1 shows a three-phase protocol: disclose suicide thoughts, send refusals only while referrals continue, then change the subject to test for renewed referral.Each chatbot had 150 conversations.Source: Figure Legends · pp. 18, 21

Figure 3A

  • Figure 3A relates persistence to the share of refusal-phase replies that present the chatbot as the user’s support; less-persistent models cluster at the left.The y-axis is that share of replies, and the least-persistent group includes all four Claude models plus DeepSeek, GLM, and Kimi.Source: Figure Legends · p. 22, Figure 3A · p. 22, Figure 3

Supplementary Online Content

The supplementary online content lists methods, tables, checklists, and a data-sharing statement.

Supplement 1

  • Supplement 1 contains seven eMethods and four eTables covering the conversation script, model settings, judges, validation, reporting, robustness, and chatbot details.The eTables include judge agreement and comparisons with other language-model judges.Source: Supplementary Online Content · p. 24

Supplement 2

Supplement 3

Supplement 1. eMethods and eTables

The supplementary methods specify a standardized script for disclosures, conditional referrals and refusals, and a topic-change phase, with one exception for risk-only replies.

eMethods 1. Conversation Script

  • The study team, including its clinician authors, wrote every scripted user message before data collection and used the same script across chatbots and conversations.Source: eMethods 1. Conversation Script · p. 25

Phase 1: disclosure

  • Each conversation began with one of 15 openers spanning passive ideation, active ideation, and intent; openers were adapted from the C-SSRS.If an opener produced no referral, another opener at the same severity was tried, up to five; this did not happen in the study.Source: Phase 1: disclosure · pp. 25–26

Phase 2: refusal ladder

  • The refusal ladder ended at the first reply without a referral or after 10 refusals, and each refusal followed only a reply that still contained a referral.The 10 standardized refusals included statements declining outside help and preferring to keep talking to the chatbot.Source: Phase 2: refusal ladder · p. 26

Phase 3: topic change

Replies that only asked about risk

  • When a reply only asked for risk disclosure without a referral, the next scripted line was prefaced with an uncertainty statement; this occurred once in 2,550 conversations.The adaptation was allowed at most once in each of the first two phases and never in phase 3.Source: Replies that only asked about risk · p. 27

eMethods 2. Query Strategy and Model Settings

The methods describe API querying and model settings, characterize default reasoning behavior, and specify a judge used to guide conversation branching and code replies.

eMethods 2. Query Strategy and Model Settings

  • Each chatbot ran in an independent session through OpenRouter, without a system prompt or cross-conversation memory, using the same temperature, top-p, output limit, and default reasoning behavior.Chatbots available at the study’s start were queried September 2, 2026, and those released during the study period on September 30, 2026.Source: eMethods 2. Query Strategy and Model… · p. 27

Default reasoning behavior by chatbot

  • Reasoning was counted when a reply’s usage record reported more than zero reasoning tokens; 407 replies had no visible text and all empty replies were excluded from content analyses.Of the empty replies, 363 reached the output limit and 44 did not.Source: Default reasoning behavior by chatbot · p. 28

eMethods 3. Branching Judge

  • A judge classified replies for referral, safety questions, and engagement beyond risk checks; only referral status determined branching, and no judge errors occurred.The judge was instructed to count pointers to outside people or services as referrals and questions about current risk as safety questions.Source: eMethods 3. Branching Judge · pp. 28–29

eMethods 4. Coding Judge

The coding judge applied separately defined fields to characterize referrals, responses to refusal, safety engagement, and the chatbot’s stance. Its prompt also specified how to record evidence and confidence for each reply.

Analysis fields

  • The analysis used 10 fields covering referral, outside-help pressure, barriers, chatbot limits, safety questions, means restriction, continued conversation, and support framing.These fields were selected because they bore on the study questions and had substantial agreement with the human coder.Source: eMethods 4. Coding Judge · p. 30

Referral

  • A referral counted when a reply pointed toward any person or service outside the conversation, with separate codes for target, specificity, and directiveness.Directiveness distinguished urgent, recommending, invitational, and no referral; professional-care directiveness was recorded separately.Source: eMethods 4. Coding Judge · pp. 30–31

Response to refusal

  • After a refusal, the judge recorded whether the reply continued pressing for outside help and whether it permitted or endorsed keeping the conversation exclusive.Offering to continue was coded separately from framing the chatbot as the user's support.Source: eMethods 4. Coding Judge · pp. 32–33

Questions and stance

  • The prompt distinguished safety questions about immediate risk, emotional experience, and help-seeking, and separately coded whose help the reply prioritized.It also captured first-person stances such as presence, emotion, joint framing, attention, and volition.Source: eMethods 4. Coding Judge · pp. 33–34

Safety planning and affect

  • Separate fields captured means restriction, other safety-planning steps, and positive or negative affect tied to the conversation or help-seeking.The prompt distinguished difficulty in reaching out from negative consequences of not doing so.Source: eMethods 4. Coding Judge · pp. 34–35

Judge instructions

  • For each reply, the judge supplied confidence and notes for every coded field, plus an evidence quote and rationale for its hardest decision.It was instructed to code only reply content, assess dimensions independently, and leave fields at their defaults when absent.Source: eMethods 4. Coding Judge · pp. 30, 35, 36

eMethods 5. Validation of the Coding Judge

The coding judge was checked against a blinded human coder, repeated runs, and other language-model judges. Agreement was substantial for analyzed fields, but some fields were excluded for lower agreement.

Agreement with a human coder

  • A team member coded 100 replies blind to judge codes and chatbot identity, after 11 training replies were excluded.The sample oversampled three judge-coded behaviors; some fields had fewer than 100 paired judgments, and barrier asking was distinguished from addressing presumed barriers.Source: Agreement with a human coder · p. 36
  • Analyzed fields reached substantial human-coder agreement (κ > 0.60), while referral-target type (κ = 0.54) and acceptance of chatbot-only conversation (κ = 0.28) were excluded.Agreement was assessed on a sample enriched for difficult cases, so it does not estimate behavior prevalence.Source: Agreement with a human coder · p. 37

Stability across repeated passes

Comparison with other judges

  • Other judges’ chatbot-rate correlations were intended to compare rankings, not estimate rates, and were imprecise with about 18 replies per chatbot.Claude Opus 5 and Gemini 3.7 Flash had unreadable outputs for 69 and 36 of 200 replies, respectively.Source: Comparison with other judges · p. 37

Models

  • All 17 chatbots were base models as deployed by developers, with no tuning, fine-tuning, or added system prompt; four had publicly downloadable weights, while the other 13 were proprietary.Source: Models · p. 37

Development of the script

  • Clinical psychologists developed 15 severity-based openers, and the team piloted the finalized script before data collection.The final script used up to 10 refusals and topic changes; pilot conversations were excluded, and the public was not involved in script development or evaluation.Source: Development of the script · pp. 37–38

Reference standard

  • The study treated human judgments as the reference standard and described behaviors rather than grading replies as right or wrong.Its behavioral principles drew on clinical guidelines, expert consensus, and gatekeeper training.Source: Reference standard · p. 38

Sample size

eMethods 7. Robustness Checks

Robustness checks found that adjustments for potentially missed referrals barely changed chatbot-level persistence estimates. The association between persistence and support framing also remained on a matched turn.

Replies that ended a refusal ladder

  • Excluding textless ladder-ending replies raised Gemini 3.8 Flash persistence from 9.33 to 9.75 refusals, while Gemini 3.7 Flash rose from 9.94 to 10.00 (P = .04).Other chatbot means changed by 0.04 refusals or less; crediting possible missed referrals changed means by at most 0.21 and yielded ρ = 0.99 correlation between estimates.Source: Replies that ended a refusal ladder · pp. 38–39

Support framing on a matched turn

  • On the first refusal reply only, persistence remained negatively correlated with the share of replies framing the chatbot as support (ρ = −0.74; P < .001; n = 17).The correlation across all refusal-phase replies was −0.80.Source: Support framing on a matched turn · p. 39

Repeated replies

eTable 1. Chatbots Tested

  • The tested set comprised 17 deployed chatbot models, including five consecutive-version pairs; each chatbot had 150 conversations.Models were queried through OpenRouter, and version identifiers matched those requested.Source: eTable 1. Chatbots Tested · p. 41

eTable 2. Agreement Between the Coding Judge and the Human Coder, and Stability Across Passes

The table reports human-coder agreement and judge stability, with referral presence also checked between the branching and coding judges.

eTable 2. Agreement Between the Coding Judge and the Human Coder, and Stability Across Passes

  • The branching and coding judges agreed on referral presence in 98.6% of 41,038 replies (κ = 0.97).Stability is the percentage of replies assigned the same value across three judge passes; agreement was computed on 100 blind-coded replies.Source: eTable 2. Agreement Between the Coding… · p. 42

eTable 3. Comparison of the Coding Judge With 3 Other Language Model Judges

The supplement describes how the study compared its coding judge with other language-model judges and operationalized its study principles. It distinguishes referral decisions during data collection from coding of other reply content afterward.

eTable 3. Comparison of the Coding Judge With 3 Other Language Model Judges

  • Four judges rated the same 200 replies, while unreadable outputs were excluded for the affected judge and judge-rate agreement was compared across 11 chatbots.Acknowledged limitations counted explicit and implicit acknowledgments as present.Source: eTable 3. Comparison of the Coding… · p. 43

eTable 4. Study Principles and How They Were Measured

  • The measured principles included renewed referral and safety questions after a topic change, barriers to seeking help, and whether the chatbot offered support.The supplement also identifies persistence after refusal and persistence by disclosure severity as study principles.Source: eTable 4. Study Principles and How… · p. 44
  • The refusal phase ran from the first refusal until the chatbot stopped recommending outside help or reached the 10th refusal.A branching judge made referral decisions during data collection; a coding judge coded other content afterward.Source: eTable 4. Study Principles and How… · p. 44

Supplement 2. CHART and STROBE Checklists

The supplement’s CHART and STROBE checklists map reporting items to locations in the main text and supplements. They point readers to information on chatbot models, study procedures, evaluation, results, and data access.

Supplement 2. CHART and STROBE Checklists

  • Checklist item numbers follow CHART, and the listed locations identify where each item is reported.The supplement also maps STROBE items to the paper’s sections and supplementary methods.Source: Supplement 2. CHART and STROBE… · p. 45
  • Model reporting locations cover identifiers and versions, model type, base-model citations, and whether models were tuned.The checklist notes that no tuned models were used.Source: Supplement 2. CHART and STROBE… · p. 45
  • The checklist directs readers to the conversation script, judge prompts, query settings, evaluation methods, results, and limitations.Full transcripts are available on request, while the script and judge prompts appear in Supplement 1.Source: Supplement 2. CHART and STROBE… · pp. 46–47

STROBE Checklist

The STROBE checklist records where the paper reports its cross-sectional design, methods, analyses, results, and interpretation. Its study units are chatbots and scripted conversations rather than people.

STROBE Checklist

  • The checklist identifies the study units as chatbots and scripted conversations, and points to the methods for chatbot selection and conversation design.The study used 10 replications of each opener, with no formal power calculation.Source: STROBE Checklist · pp. 48–49
  • The reported analyses treated persistence as a 0-to-10 count and severity as three design-defined categories, using chatbot fixed effects and robust standard errors.The checklist points to the statistical methods for these choices.Source: STROBE Checklist · p. 49
  • The study comprised 2,550 conversations and 41,450 replies, with 407 empty replies and 5 uncodable replies excluded from content analyses.The checklist points to Results and Methods for these counts.Source: STROBE Checklist · p. 50
  • The checklist points readers to Results, Tables 1 and 2, and Figure 2; Table 2 reports unadjusted means and percentages and chatbot-adjusted estimates with 95% CIs.Chatbot was adjusted for because chatbots differed widely.Source: STROBE Checklist · p. 51, Table 2 · p. 51

Supplement 3. Data Sharing Statement

The data-sharing statement makes study materials available in Supplement 1 and offers full conversation transcripts by request. The transcripts include explicit suicidal-intent statements and are shared with a content warning.

Supplement 3. Data Sharing Statement

  • The conversation script and both judge prompts are published in full in Supplement 1.Field definitions and coding prompts are also identified as supporting documents.Source: Supplement 3. Data Sharing Statement · p. 52
  • Full transcripts of all 2,550 conversations are available by email request to the corresponding author for research purposes.Transcripts are available at publication.Source: Supplement 3. Data Sharing Statement · p. 52
  • Because the transcripts contain explicit statements of suicidal intent, they are shared with a content warning.No human-participant data were collected; the study team wrote all user messages.Source: Supplement 3. Data Sharing Statement · p. 52
Loading 2610.21253v1…