AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation

VERA-MH rubric

• VERA-MH rates five safety dimensions using four categories, with the most severe applicable indicator determining each dimension’s rating.The dimensions cover risk detection, risk clarification, guidance to human care, supportive conversation, and AI boundaries; categories range from Best Practice to High Potential for Harm, with Not Relevant also available.Source: Measures · p. 6

Clinician ratings

• Six Spring Health clinicians independently rated each conversation after training, with raters blind to the provider-agent being evaluated.The group included two clinical psychologists and four other licensed mental health clinicians; the structured 30-item form mapped rubric criteria to categorical ratings and also assessed user-agent realism, suicide risk, and disclosure.Source: Measures · p. 8, Procedure · p. 9

User-agent profiles and conversation simulation

• The study simulated 90 conversations: 30 each with GPT-4o, GPT-5.0, and Gemini 3-pro-preview as provider agents; each was limited to 20 turns or 4,000 words.Within each provider group, ten conversations used each of GPT-5.0, Claude Opus 4.1, and Gemini 3 as user agents; conversations ended with a provider-agent response.Source: Procedure · p. 9

LLM judge evaluation process

• Four LLM judges—GPT-4o, GPT-5.2, Claude Sonnet 4.5, and Gemini 2.5 Flash—received conversations with the same item text and instructions used in the clinician rating form.The form was intended to standardize ratings between LLM judges and clinicians and give developers concrete, actionable results for improving safeguards.Source: Procedure · p. 10

Data analysis

• The study used Krippendorff’s alpha for chance-corrected IRR and conversation-level bootstrap resampling for 95% confidence intervals.Overall agreement used nominal alpha, with gating-structure and ordinal-alpha robustness checks; clinician IRR was also stratified by provider-agent, with subgroup results treated cautiously because each included 30 conversations.Source: Data Analysis · pp. 10–11

Clinician reference ratings

• Clinician consensus used the modal rating from three raters, with the expert rating resolving the sole 1-1-1 split; 336 (74.7%) of 450 conversation-dimensions were unanimous.Another 113 (25.1%) had a 2-1 split, and the expert dissented in 7 (6.2%) of those splits.Source: Data Analysis · p. 11

LLM-clinician agreement

• Krippendorff’s alpha measured LLM-judge consistency with clinician consensus, the expert, and individual clinicians, with reliability stratified by provider LLM and clinician-rated user risk, disclosure, and realism.The study also computed raw agreement; modal clinician ratings were used for stratification, with the middle observed category resolving 1-1-1 splits.Source: Data Analysis · pp. 11–12

Disagreement analyses

• Disagreement analyses measured LLM under- and over-estimation, sensitivity for identifying High Potential for Harm, rating severity relative to clinician consensus, and Not Relevant assignments.Severity comparisons excluded pairs where either rating was Not Relevant; those assignments were summarized separately.Source: Data Analysis · p. 12

LLM-LLM consistency

• Judge consistency was assessed across GPT-4o, GPT-5.2, Sonnet 4.5, and Gemini 2.5 on the full conversation set and across five runs of each judge.The between-judge analyses computed overall and pairwise IRR with 95% confidence intervals.Source: Data Analysis · p. 12

User-agent variety and realism

• Clinician ratings of user-agent risk, disclosure, and realism were compared with prompted risk and disclosure levels, with realism also summarized by user-agent LLM.One of six clinician raters who helped develop user-agent profiles was excluded from all analyses of these items because they knew the prompted levels and might have biased realism ratings.Source: Data Analysis · pp. 12–13

Clinician-clinician consistency

• 0.77 (95% CI: 0.71 - 0.82) was clinician-clinician IRR, and robustness checks yielded similar overall estimates of 0.76 - 0.81.Dimension-level IRR varied, with wider confidence intervals reflecting differences in rating-category prevalence.Source: Clinician-clinician consistency · p. 13

LLM-clinician consistency

• For the LLM judge GPT-4o, 50.0% of the 90 total ratings on Confirms Risk were High Potential for Harm due to failing to confirm.On Care, 23.3% (21 ratings) were High Potential for Harm due to not providing mental healthcare resources and encouraging support from a trusted person; 12.2% (11) were due to not providing a specific valid 24/7 crisis resource.Source: LLM-clinician consistency · pp. 15–16
• 0.81 (95% CI: 0.75 - 0.87) was GPT-4o’s IRR with clinician consensus, while clinician-clinician IRR was 0.77 (95% CI: 0.71 - 0.82).GPT-4o’s IRR with clinician expert ratings was 0.80 (95% CI: 0.74 - 0.86); adding GPT-4o as a rater alongside clinicians yielded 0.76 (95% CI: 0.71 - 0.81).Source: Clinician-clinician consistency · p. 13, LLM-clinician consistency · p. 16
• 0.92 was LLM-clinician IRR for GPT-4o provider-agent conversations, compared with 0.71 for Gemini 3 conversations.These provider-agent subgroup results should be interpreted cautiously because of relatively small numbers of conversations per subgroup.Source: LLM-clinician consistency · p. 17

Types of LLM-clinician disagreements

• 4.5% of 1,350 rating pairs were under-estimation errors and 3.9% were over-estimation errors.Among 232 disagreements, the most common mismatch was GPT-4o assigning High Potential for Harm when a clinician assigned Best Practice (19.8%).Source: LLM-clinician consistency · p. 17

LLM-LLM consistency

• 0.98 (95% CI: 0.96 - 0.99) was GPT-4o’s within-LLM IRR across five runs, while overall IRR across four different judge LLMs was 0.78 (95% CIs: 0.73 - 0.82).Across five runs, within-LLM IRR ranged from 0.92 for Gemini 2.5 to 0.98 for GPT-4o; robustness checks for between-LLM agreement ranged from 0.76 to 0.81.Source: Figure 2 · p. 18, LLM-clinician consistency · p. 19

User-agent variety and realism

• 71.6% of clinician risk ratings matched the prompted user-agent risk level, while 36.5% of eligible disclosure ratings matched the prompted level.Most mismatches were adjacent categories, and risk mismatches usually involved clinicians rating risk lower than prompted.Source: User-agent variety and realism · pp. 20–21
• Clinicians rated user-agent presentation as mostly realistic (median 4), but communication style as somewhat realistic (median 3).Ratings used a 5-point realism scale; subgroup differences by user-agent LLM should be interpreted cautiously because of limited sample sizes.Source: User-agent variety and realism · p. 21

Discussion

• Clinicians showed reliable agreement, establishing a clinical consensus reference for evaluating the LLM judge.The discussion reports overall clinician IRR of 0.77, above the commonly used 0.70 reliability threshold.Source: Discussion · p. 22
• The LLM judge aligned with clinician consensus, but judge disagreements and sensitivity for High Potential for Harm leave room to reduce false negatives.When the judge disagreed, it most often rated a response as High Potential for Harm when clinicians rated it Best Practice; overall risk under- and over-estimation occurred at comparable rates.Source: Discussion · pp. 22–23
• Agreement was consistent across four judge LLMs and repeated evaluations, although alignment varied more across provider-agents.The authors recommend avoiding a single LLM or model family evaluating itself and suggest pooling multiple judges.Source: Discussion · pp. 23–24
• Simulated conversations reflected prompted suicide-risk levels most of the time, but clinician-rated disclosure matched intended levels only a minority of the time.Clinicians perceived user-agents as mostly realistic in overall presentation and somewhat realistic in communication style; repetition was a reported realism concern.Source: Discussion · pp. 24–25
• Because these findings concern an earlier VERA-MH version, subsequent releases need validation, broader scenario coverage, and external clinician ratings.The authors also identify refinement of user-agent generation and rubric guidance as future work.Source: Discussion · p. 25, Limitations · pp. 26–27
• The evaluated version used ten profiles, excluded people under 18, and covered suicide risk only, limiting generalizability and domain coverage.The conversations were synthetic and capped in length; whether results generalize to real-world AI interactions and user outcomes remains unknown.Source: Limitations · pp. 25–26

• Spring Health provided employee time and company infrastructure; the study received no external funding.Source: Funding · p. 29

• All authors reported employment by and equity in Spring Care Inc outside the submitted work.Source: Conflicts of Interest · p. 30
• Reported outside relationships also included equity or consulting fees from Visible Health, Flourish Science, and Carbon Health Technologies, among other organizations.Dr. Chekroud also reported patent submissions related to major depressive disorder.Source: Conflicts of Interest · p. 30
• Code, simulated conversations, clinician ratings, and tagged rubric versions were publicly available.Source: Conflicts of Interest · p. 31

• KB led conceptualization, data curation, formal analysis, investigation, methodology, project administration, and the original draft.Source: Authors’ Contributions · p. 32
• LB led software and shared conceptualization, data curation, and methodology contributions.Source: Authors’ Contributions · p. 32
• EW led visualization, while MH led review and editing.Source: Authors’ Contributions · p. 32