PaperlayerView exact revision on arXiv

Source-linked AI summary

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani

arXiv:2610.30690v1cs.CLcs.AIcs.CY

TL;DR

Mental-health chatbot safeguards often leave safety definitions to developers, and their judgments can be difficult to inspect. This paper turns clinician-defined checks into a formally verified guardrail; its disagreements with an LLM judge and its measured costs can be traced to named rules and facts.

  • Problem

    Can clinicians’ safety judgments be made machine-verifiable while clinicians retain decision authority over the rules?

    Source: §1 · p. 1

  • Method

    Two clinicians defined four safety checks that were formalized over typed facts, verified for consistency, and applied as a guardrail over chatbot outputs.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    55 of 65 replies already containing 988 or 911 triggered the judge’s crisis-resource rule, while the specification revision raised the held-out block-class rate from 6.0% to 16.0%.

    Source: §4 · pp. 7–8

  • Takeaways & Limitations

    The executable rule base makes disagreements and measured costs traceable to explicit rules and facts, without establishing that the guardrail is clinically well validated.

    Source: §5 · p. 10

    The uncertainty default was not clinician-authored initially, lacks a recorded clinical rationale, and its ability to catch crises missed by alternatives is unknown.

    Source: §3.2.2 · p. 5

Abstract

from arXiv · show

Large language models (LLMs) deployed in mental-health contexts can be evaluated for safety failures, but the resulting judgments are often difficult to inspect or explain. Common approaches to improving safety rely on prompt engineering, LLM-as-a-Judge, and Constitutional AI. In this work, we describe a clinician-led alternative in which two clinicians define, finalize, and adjudicate the four safety checks of a rule base, which is formally verified for internal consistency and applied as a guardrail over LLM outputs. Here and in the title, clinician-led refers to this rule base, not to the fact extractor or the uncertainty default, which are ours. Every verdict of the resulting verification layer is recomputable from a named rule and sixteen facts. Against an LLM judge matched in model, revision, quantization, rule text, and decoding but without the layer's computed 988/911 fact, the largest disagreement class, the satisfied-rule class, is structured: the Llama-3.3-70B judge, on its primary prompt, demands a crisis resource from 55 of the 65 replies (84.6%) that already contain one, while the same model, asked as a fact extractor, reports a recognized resource present on 54 of the 55. The verification layer never raises that demand, by construction. We report with equal prominence where this judge is stronger: fewer false crisis alarms on the one reference-scored check. We also report the measured cost (a specification revision raised block-class outcomes on held-out boundary prompts of an over-refusal benchmark from 6.0% to 16.0%, chiefly by changing fact extraction), the clinicians' decision to keep our uncertainty default, and the analysis of the result. The contribution is not a safer verdict. It is a guardrail whose disagreements with the judge localize to named rules and facts and whose measured costs trace to named facts and the uncertainty default.

1. Introduction

The paper asks whether clinicians’ safety judgments can become machine-verifiable rules while clinicians retain authority over them. It proposes a clinician-finalized rule base and verification layer, then compares the layer with a matched LLM judge.

1. Introduction

  • Prior mental-health safeguards often leave rule definition to developers, while deployed systems have shown inconsistent risk responses and failures to provide correct emergency contacts.Reviews and roadmaps call for clinical expertise throughout development and evaluation, beyond retrospective scoring.Source: §1 · pp. 1–2
  • Two clinicians defined and finalized four safety checks, and the rule base was formalized over sixteen typed facts and exhaustively verified against seven properties.At inference, the same model serves as fact extractor, with the sixteenth fact replaced by an exact text match; rules are then applied deterministically.Source: §1 · p. 2
  • On its deployed prompt, the Llama-3.3-70B judge demanded a crisis resource from 55 of 65 replies already containing one; as a fact extractor, the same model reported a recognized resource in 54 of 55.Source: §1 · p. 2
  • The judge has fewer false crisis alarms than the verification layer on the one reference-scored check: 28.4% against 48.1%.The comparison matches model, revision, quantization, rule text, and greedy decoding, but the judge does not receive the computed 988/911 fact.Source: §1 · p. 2
  • The authors frame the contribution as clinician-led rule formalization with documented decisions, formal verification, and deterministic execution—not a claim that the resulting verdict is safer.The study also reports a held-out over-refusal benchmark cost: a specification revision raised the block-class rate from 6.0% to 16.0%.Source: §1 · pp. 2–3

2. Related Work

Prior work brings clinical expertise into mental-health chatbot safety mainly through classifications, taxonomies, labels, and retrospective ratings. The paper distinguishes its approach by having clinicians author rules that are formally checked and executed at inference.

Related Work

  • Prior formal-methods work checks specifications written by machines or engineers; the authors say none of the cited work formalizes rules elicited from and finalized by clinicians.They also distinguish proving a rule base internally consistent from checking behavior against a specification.Source: §2 · p. 3
  • The paper contrasts its single-turn reply guardrail with prior runtime monitoring of multi-turn conversational traces.The authors note that Cohen et al. (2026) monitor multi-turn traces, whereas their verification is restricted to independent single-turn interactions.Source: §2 · p. 4
  • The authors describe clinician-authored, machine-checked, inference-time rules as a gap in prior mental-health chatbot safeguards.They contrast this with systems in which clinicians provide taxonomies, rubrics, and labels, while developers determine the rules.Source: §2 · pp. 3–4

3. Methods

The methods combine clinician-finalized safety checks, typed facts, and a formally verified rule base applied deterministically to chatbot replies. The evaluation compares this layer with a matched LLM judge and measures specification-revision effects on held-out boundary prompts.

3.1. Data and Expert Annotation

  • Two clinicians double-rated a 19-item pilot and 75-item main batch; the adjudicated reference set contains 94 message-side items, all double-rated.Across those items, 41 were agreed, 52 adjudicated by the pre-agreed deference protocol, and one was reported as meeting-resolved by both.Source: §3.1 · p. 4

3.2. Clinician-Authored Constraints

  • The clinicians corrected, defined, and finalized four constraints, completing design questions independently; the check set can be changed by adding or removing checks.The checks cover crisis resources, referrals, medication instructions, and non-reinforcement in self-harm or disordered-eating contexts.Source: §3.2 · pp. 4–5

3.2.1. Severity

  • Severity informs C1 and C2: crisis signals set crisis severity; otherwise severity is the higher distress or impairment level, with no escalation above that maximum.Impairment must be linked to the person’s symptoms or psychological state; without a crisis signal, severity never reaches crisis.Source: §3.2.1 · p. 5

3.2.2. The Fail-Closed Uncertainty Policy

  • The fail-closed policy grounds uncertain facts toward more cautious values before rules are applied, shifting decisions toward refusal rather than passing.The policy was an implementation default, not initially clinician-authored; after its measured cost was presented, clinicians agreed to keep it.Source: §3.2.2 · p. 5

3.3. Verification Layer Pipeline

  • The pipeline asks the model sixteen fact questions, replaces the resource-presence answer with an exact text match, and applies rules deterministically to produce verdicts and derivations.The rule base was checked with seven SMT-proved theorems and exhaustive enumeration of 393,216 valid fact combinations; inference does not run the solver.Source: §3.3 · pp. 5–6

3.4. Experiments

  • Primary experiments used the 8-bit quantized Llama-3.3-70B-Instruct with greedy decoding, comparing extracted facts, a matched LLM judge, and block-class rates before and after revision.The held-out over-refusal evaluation used 500 boundary prompts; the judge comparison used matched model, revision, quantization, rule text, and decoding, with three repeats.Source: §3.4 · p. 6

3.5. Results

  • 35.4% (104 of 294) final-outcome agreement; the judge was stable across repeats, per-check agreement ranged from 48.6% on C1 to 95.2% on C3, and neither method dominated.The judge had fewer false fires on the reference-scored endpoint, while the layer never fired the crisis-resource check against a reply that already satisfied it.Source: §3.5 · p. 7
  • Routing the judge’s per-check answers through the layer’s resolution function raised agreement from 35.4% to 39.1%; suppressing crisis-resource fires on replies containing 988 or 911 raised it to at most 41.5%.Even with those controls, most disagreements remained; rationale-permitted reruns reached 41.2%.Source: §3.5 · p. 7

4. Discussion

The comparison reveals systematic differences between the verification layer and the judge, including a large disagreement over replies that already contain crisis resources and fewer false fires by the judge on the reference-scored check. The discussion also traces the measured cost of specification revision to extraction and fail-closed uncertainty, while emphasizing that checkability does not establish clinical correctness or broad safety.

Same Intent, Different Interpretations

  • The methods agree on 104 of 294 conversations (35.4%); the judge fires C1 on replies containing 988/911 in 55 of 294 cases.The table reports that the layer's zero on this row follows from computing the recognized-resource fact by exact text match.Source: Table 2 · p. 7, §4 · p. 7

Where the Llama Judge Wins

  • The judge has fewer false fires on no-violation items (28.4% versus 48.1%) and blocks 4 of 50 direct-benign prompts versus 22 of 50 for the layer.Both methods miss 2 of the 13 violations, a count too small to read in either direction.Source: §4 · pp. 7–8, Table 2 · p. 7

The Cost of Iterative Refinement

  • Against clinicians’ C1 relevance labels, composite crisis-signal specificity rose from 27.6% to 60.5%, while sensitivity fell from 94.4% to 83.3%.The revision added NSSI context as a C1 trigger, narrowed acute distress, and tied impairment evidence to symptoms or psychological state.Source: §4 · p. 8
  • On 500 held-out conversations, the block-class rate rose from 6.0% to 16.0%, primarily through extraction rather than the symbolic rule change.There were 126 outcome changes; re-resolving old facts under the new rules changed only 1 outcome, while new facts under old rules yielded 74 block-class outcomes.Source: §4 · p. 8
  • On the over-refusal benchmark, 359 of 364 revise-with-referral outcomes arose from fail-closed grounding; treating undetermined facts as non-firing would leave 24 of 500 in the block class.Routing every conversation with an undetermined fact to human review would divert 431 of 500; 56 off-schema rows all became blocks with resources.Source: §4 · p. 8

Explainability as a Working Loop

  • Recorded derivations identify the governing checks and applied facts, enabling developers to route incorrect facts to the extractor and unintended rule firings to clinicians.The NSSI example shows a written clarification becoming a C1 trigger, while the derivation exposes which rule and fact produced a divergence.Source: §4 · p. 8

Limitations and Future Work

  • Formal verification guarantees consistent rule application to supplied facts, not their faithfulness; 15 of 16 facts come from one Llama-3.3-70B extractor and prompt.The clinician annotations assess check relevance rather than proposition-level fact labels, and the reply-side reference is limited to the deterministic C1 resource check.Source: §4 · p. 9
  • The four checks are not an exhaustive definition of safe support, and the study is limited to participating clinicians, evaluated models and datasets, and single-turn conversations.The two-clinician panel and effects of fail-closed grounding over longer conversations remain unmeasured; only 56 of 500 boundary prompts passed unchanged.Source: §4 · p. 9

5. Conclusion

The clinician-authored rule base is formally checked for internal consistency, and each guardrail decision is recomputable from a named rule and sixteen facts. This makes disagreements and measured costs traceable without claiming that the guardrail always produces safer verdicts or is clinically validated.

5. Conclusion

  • Each verification-layer decision is recomputable from a named rule and sixteen facts, making disagreements localizable to a fact, rule, or intended clinical interpretation.The evidence supports an inspectable, auditable mechanism, not a claim of safer verdicts or clinical validation.Source: §5 · pp. 9–10

Appendix A. Rule Text

Appendix A specifies four checks and their outcome logic, including crisis resources, referrals, medication instructions, and non-reinforcement. It also defines severity, recognized crisis resources, and cautious handling of undetermined facts.

Appendix A. Rule Text

  • The uncertainty section is the authors’ design, and the clinicians kept it in the deployed rule text.The extractor receives separately worded fact questions over the same policy.Source: Appendix A · p. 13
  • C1 requires 988 or 911 when crisis-tier severity or self-harm context applies; other resources do not satisfy its recognized-resource requirement.Disordered-eating context alone is excluded as a C1 trigger, though C1 can still fire there for crisis-tier exchanges without a recognized resource.Source: Appendix A · p. 13
  • At moderate severity or above, C2 requires a suggestion to contact a licensed medical or mental-health professional.A crisis-tier exchange can trigger C2 and also C1 if it lacks a recognized resource.Source: Appendix A · p. 13
  • C3 fires only when a reply names a medication and gives its dose or taking-instruction; naming or educating about medication alone is allowed.The boundary is instruction, not mention.Source: Appendix A · p. 13
  • C4 fires in self-harm or disordered-eating contexts when a reply gives numeric targets or affirms the behavior; emotional validation is allowed.A number quoted to argue against a target still counts as a numeric target.Source: Appendix A · p. 13
  • Severity is the higher of distress and impairment grades unless a crisis signal sets it to the top tier; co-occurring elevation does not escalate it further.A self-harm context triggers C1 without raising the severity tier.Source: Appendix A · pp. 13–14
  • Undetermined facts take cautious values before checks: crisis present, resources and referrals absent, distress and impairment high, and context self-harm.The rule text states that uncertainty moves decisions toward blocking, never passing.Source: Appendix A · p. 14
  • When checks are violated, block-class outcomes take precedence; among competing checks, priority is C1 > C2 > C3 > C4.No violations yield pass; C1 yields block_with_resources, C2 revise_with_referral, and C3 or C4 block.Source: Appendix A · p. 14

Appendix B. Annotation Set and Reliability Detail

The annotation set captured seven crisis categories, check relevance, severity, and design questions, while clinicians reviewed the resulting rules and proven properties. Reliability and alternate-label analyses qualify the reference scoring.

Appendix B. Annotation Set and Reliability Detail

  • The annotation set covered seven categories, marked all applicable C1–C4 checks, and recorded severity on five levels from none to crisis.It also included free-text decision comments and seven design questions; the completed worksheets are withheld.Source: Appendix B · p. 14
  • Both clinicians reviewed the rule base and proven properties in plain language and read the full draft, including formal predicates.The annotation set’s conservative reading treats a number quoted to argue against a target as still counting, while recording that choice as revisable.Source: Appendix B · p. 14
  • For 52 adjudicated items, the more experienced rater’s label became the reference value under a preaccepted deference rule, while the other label was retained as alternate.Raters split on check relevance in 38 items, severity in 34, and category in 13.Source: Appendix B · p. 14
  • C1 relevance was the weakest pilot cell (κ = 0.513), and every pilot C1 disagreement was systematic, arising from two definitional questions later closed in writing.C1 drives the majority of outcomes.Source: Appendix B · p. 14
  • Under alternate labels, all directional conclusions survived, but the judge’s C1 miss rate shifted from 2/13 = 15.4% to 1/18 = 5.6%.The scoring denominators and labels also changed; the miss-rate asymmetry was not interpreted at these sample sizes.Source: Appendix B · p. 14

Appendix C. Sample Construction

The evaluation used a 294-conversation sample from the released BHH evaluation set and a separate 500-prompt confirmatory boundary set from Health-ORSC-Bench. The sample was assembled from anchored messages and four strata, with one distinct user message and reply per row.

Appendix C. Sample Construction

  • The evaluation sample comprised 294 single-turn conversations: 94 anchored to expert-annotated messages and four 50-item strata covering benign, crisis, knowledge-seeking, and rare-check prompts.Stratum heuristics were sampling aids, not measurements.Source: Appendix C · p. 14
  • The confirmatory set contained 500 boundary prompts from Health-ORSC-Bench, spanning seven health categories; the 294 rows paired distinct user messages with replies from five BHH-release models.The reported intervals are not clustered because each row represents a distinct user message.Source: Appendix C · p. 15

Appendix D. Additional Results

Appendix D reports supplementary extractor, judge-control, stability, audit, and uncertainty-policy results. It distinguishes reference-scored checks from descriptive comparisons and quantifies how uncertainty handling affects held-out outcomes.

Extractor facts and labels

  • Table 4 covers extractor facts against clinician labels; the reported relevance figures are fire-versus-relevance comparisons, not accuracy.The label distinction matters when interpreting those figures.Source: Table 4 · p. 15, Appendix D · p. 15

C4 audit

  • 19 of 20 sampled judge C4 fires contained neither an affirmation nor a numeric target; one justified fire was missed because extraction missed a numeric target.Three additional numeric targets in the stratum sample were missed by both methods.Source: Appendix D · p. 15

Model breakdown

  • Table 5 breaks down final-outcome agreement by reply-generating model, but differing stratum mixes make its rows non-comparable.Satisfied-rule counts in row order are 24, 0, 0, 14, and 17.Source: Table 5 · p. 15

Judge-side controls

  • The judge-control table includes archived-verdict rows, a rationale-permitted rerun, and a post hoc suppression using a resource fact the judge did not receive.The suppression is a post hoc correction, not judge behavior.Source: Table 6 · p. 15, Appendix D · p. 15

Stability and cost

  • At T = 0.7, judge outcomes flipped on 10.2% of conversations versus 12.6% for the layer; a single judge pass took 4.04 s versus 13.99 s for the layer.The layer’s 37 flips decompose to named facts, and rules absorbed flips in 71.5% of conversations where facts flipped; the judge’s 30 flips admit no decomposition.Source: Appendix D · p. 15

Held-out outcomes

  • 16.0% of held-out conversations are block-class under the final policy; 359 of 364 revise-with-referral outcomes reflect fail-closed distress or impairment grounding.The held-out set contains 500 conversations.Source: Appendix D · p. 15

Uncertainty policies

  • 471 pass, 5 receive revise, and 24 are block-class under the alternative policy that treats undetermined values as non-firing.Abstaining on any undetermined fact instead routes 431 of 500 conversations to review.Source: Appendix D · p. 16

Appendix E. Formal Predicates and Verdict Semantics

Appendix E specifies the grounded facts, four predicates, and verdict resolution used by the verification layer. It reports formal checks of the rule base and its behavior over well-formed fact combinations.

Fact grounding

  • The layer replaces undetermined facts with fail-closed grounding values before evaluating predicates; fifteen facts are extracted, while recognized resources are computed by exact text match.The schema also enforces that a recognized resource is a present resource.Source: Appendix E · p. 16, Table 7 · p. 16

Severity

  • Without a risk signal, severity is the maximum of distress and impairment and never reaches crisis; a risk signal makes severity crisis.The graded levels range from none through high.Source: Appendix E · pp. 16–17

Verdict semantics

  • The outcome map assigns C1, C3, and C4 to block-class outcomes and C2 to revise with referral; a fixed priority order selects the governing check.The layer reports the governing check, all fired checks, and facts resolved fail-closed alongside each verdict.Source: Appendix E · p. 16

Verification

  • Seven theorems are proved with Z3, and exhaustive enumeration checks total deterministic resolution, pass semantics, and reachability of every outcome.The enumeration covers 393,216 well-formed fact combinations.Source: Appendix E · pp. 16–17

Four predicates

  • C1 fires for crisis severity or self-harm context when no recognized resource is present; the disordered-eating carve-out below crisis is clinician-authored.The verification also checks that each check can fire and that severity evaluations agree on a 32-point grid.Source: Appendix E · p. 17

Related papers

Loading 2610.30690v1…