When Retrieval Helps: Selective Retrieval for Single-Turn Mental-Health QA

• Mental-health queries can combine distress, symptoms, treatment concerns, and coping requests, requiring empathetic and specific responses without unsafe guidance.Source: §1 · p. 1
• Retrieved evidence may be generic, weakly related, or overly directive, so Always Retrieval can improve specificity while introducing noise and safety risks.Source: §1 · p. 1
• Existing adaptive RAG criteria such as complexity and confidence do not fully capture needs for safety grounding, coping guidance, or psychoeducation.Source: §1 · p. 1
• The proposed policy treats retrieval as a domain-specific control problem using psychoeducation, coping support, response specificity, and a hard safety trigger.Source: §1 · p. 1
• The controlled comparison evaluates Closed-book, Always Retrieval, and Selective Retrieval with the same domain-adapted generator.Source: §1 · p. 2

• Mental-health QA requires balancing empathy, specificity, factual caution, and professional boundaries beyond ordinary answer accuracy.Source: §2.1 · p. 2
• RAG combines parametric knowledge with external evidence, but noisy contexts can make retrieval less reliable than closed-book generation.Source: §2.2 · p. 2
• Adaptive retrieval methods commonly use query complexity, self-reflection, information need, utility criteria, or uncertainty.Source: §2.2 · p. 2
• This work instead defines retrieval need through psychoeducation, coping support, response specificity, and safety, evaluating the policy on standard and adversarial mental-health QA.Source: §2.2 · p. 2
• Psychotherapy-grounded systems such as coTherapist use authoritative therapy and clinical resources rather than unrestricted web evidence.Source: §2.3 · p. 2

• The task is to generate a supportive response y for a user query q in single-turn mental-health QA.Source: §3 · p. 2
• The generator is domain-adapted on MentalChat16K, a benchmark of synthetic and anonymized counseling-related question-answer pairs.Source: §3 · p. 2
• At inference, the model may optionally retrieve supporting evidence from a small corpus of authoritative mental-health resources.Source: §3 · p. 2

4.1 Fine-Tuning Base Generator

• QLoRA adapts a single instruction-tuned generator on MentalChat16K, and that generator remains fixed across all retrieval conditions.Source: §4 · p. 2, §4.1 · pp. 2–3
• Figure 1 organizes the framework into generator adaptation, BM25 indexing of a source-typed corpus, and inference-time selective retrieval.Source: §4 · p. 2, Figure 1 · p. 3

4.2 Guideline Corpus Construction

• The guideline corpus follows psychotherapy-grounded retrieval practice and organizes evidence into coping, psychoeducational, and safety resources.Source: §4.2 · p. 3
• The corpus contains 40 documents covering coping strategies, psychoeducation about common conditions, crisis guidance, urgent help-seeking, and medication caution.Source: §4.2 · p. 3
• Documents are cleaned, segmented into overlapping word-level chunks, and stored with source-family and document metadata.Source: §4.2 · p. 3
• BM25 retrieves top-k chunks from the corpus, with k=3 in the main experiments, keeping the setup focused on activation policy rather than retriever optimization.Source: §4.2 · p. 3

4.3 Inference-time Selective Retrieval

• The three inference policies are Closed-book, Always Retrieval, and Selective Retrieval.Source: §4.3 · p. 4
• Selective Retrieval first generates a closed-book draft, then scores the query and draft for information, coping, and specificity needs before deciding whether to retrieve.Source: §4.3 · p. 4

Experimental Design

• The controlled comparison evaluates Closed-book, Always Retrieval, and Selective Retrieval under the same tuned generator.This isolates retrieval-policy effects from generator differences.Source: §5.1 · p. 4, §5 · p. 4

Main Results

• Always Retrieval improves specificity over Tuned Closed-book but lowers overall quality and raises the medical-advice flag rate.The trade-off appears within the tuned variants rather than in comparison with the Base LM reference.Source: §5.1 · p. 4
• Selective Retrieval improves Overall and Empathy over Tuned Closed-book while preserving a zero medical-advice rate.Its benefit is characterized as a quality-safety trade-off rather than a large average-score gain.Source: §5.1 · p. 4

Safety Stress Test

• Always Retrieval increases macro failure rate on CounselBench-Adv, especially through therapy- and assumption-related failures.CounselBench-Adv directly probes targeted unsafe or undesirable behaviors.Source: §5.2 · p. 5
• Selective Retrieval matches the shared closed-book macro failure rate while activating retrieval for only 7.5% of adversarial questions.This limits the additional failures associated with unconditional retrieval under safety stress testing.Source: §5.2 · p. 5

Expert Audit

• The expert audit provides a targeted qualitative check rather than clinical validation of the automatic evaluation pattern.It focuses on professional boundaries, overly directive advice, and practical helpfulness.Source: §5.3 · p. 5

Threshold Calibration

• The selective policy uses a mean retrieval-need threshold and a high-axis route threshold to control activation.The mean threshold aggregates information, coping, and specificity needs, while the route threshold responds to strongly expressed informational or coping needs.Source: §5.4 · p. 5
• Retrieval activation drops sharply at τ= 2.5 and remains near the hard-safety floor at larger thresholds; the main setting uses τ= 3.25.The sweep indicates that higher thresholds mainly preserve safety-triggered retrieval and a small number of high-need cases.Source: §5.4 · p. 5

Conclusion

• Always Retrieval improves specificity but can introduce additional safety-sensitive failures through overly directive, clinical, or medicalized guidance.Selective Retrieval instead preserves closed-book behavior for low-need cases and activates evidence under explicit utility or safety triggers.Source: §6 · p. 5
• Selective Retrieval’s central benefit is controlling degradation introduced by unconditional retrieval.The paper presents retrieval activation as a control problem rather than assuming retrieval is beneficial for every query.Source: §6 · p. 5

Limitations

• The gate combines fixed safety patterns with utility scores produced by the same generator used for response generation.Independently calibrated or learned policies may improve robustness.Source: §6 · p. 5
• Validation uses one open-source generator family and two splits from the same benchmark framework, limiting claims of generality.The authors call for evaluation across model families and independently constructed mental-health datasets.Source: §6 · p. 5
• The compact 40-document corpus improves source controllability but limits evidence coverage, while evaluation relies primarily on LLM judges and a small expert audit.These design choices constrain evidence breadth and the extent of human reliability checking.Source: §6 · p. 5
• The single-turn setting does not capture longitudinal context, evolving retrieval needs, or multi-turn repair behavior.Future work includes temporally grounded retrieval for multi-session interactions.Source: §6 · p. 5

Future Work

• Future work extends evaluation to multiple generator families, independent datasets, larger evidence collections, learned gates, hybrid retrieval, and larger expert evaluations.These directions aim to separate retrieval-policy effects from model-family and dataset-composition effects.Source: §6 · p. 5

• MentalChat16K provides synthetic counseling QA pairs and anonymized intervention transcripts for generator adaptation.Source: §A.3 · p. 8
• CounselBench-Eval contains 100 real patient questions with clinically grounded response-quality dimensions.Source: §A.3 · p. 8
• CounselBench-Adv contains 120 expert-authored adversarial questions targeting six mental-health failure categories.Source: §A.3 · p. 8
• The MentalChat16K-adapted QLoRA adapter and implementation code are publicly available.Source: §A.3 · p. 8

Threshold comparison

• Lowering τ from 3.25 to 2.25 increases retrieval activation from 9.0% to 38.0% on CounselBench-Eval and from 7.5% to 43.3% on CounselBench-Adv.Source: §A.1 · p. 8
• On Eval, overall quality and specificity slightly decrease under the lower threshold.Source: §A.1 · p. 8
• On Adv, macro failure increases from 0.0250 to 0.0417, mainly because of assumption failures.Source: §A.1 · p. 8

Conclusion

• The results support τ=3.25 as the conservative operating point.Source: §A.1 · p. 8

Generation and routing

• The pipeline fixes one MentalChat16K-adapted generator across Closed-book, Always Retrieval, and Selective Retrieval conditions.Source: §A.2 · p. 8
• Selective Retrieval generates a closed-book draft, scores information, coping, and specificity needs from 1 to 5, then either returns the draft or regenerates with evidence.Source: §A.2 · p. 8

Corpus and retrieval

• BM25 retrieves top-k=3 evidence, while the corpus, retrieval depth, generation settings, and judging scripts remain fixed in threshold ablations.Source: §A.2 · p. 8
• The guideline corpus contains coping, psychoeducation, and safety resources segmented into overlapping chunks for indexing.Source: §A.2 · p. 8

Evaluation

• CounselBench-Eval reports Overall, Empathy, Specificity, and Medical Advice Yes Rate, whereas CounselBench-Adv reports six failure rates and their macro average.Source: §A.2 · p. 8
• A targeted expert audit provides a qualitative reliability check for whether automatic retrieval-related patterns are clinically plausible.Source: §A.2 · p. 8