PaperlayerView exact revision on arXiv

Source-linked AI summary

When Retrieval Helps: Selective Retrieval for Single-Turn Mental-Health QA

Hyunseo Oh, Chong-Kwon Kim, Yoonhyuk Choi

arXiv:2609.03454v1cs.CLcs.IR

TL;DR

Single-turn mental-health QA requires retrieval decisions that account for support needs and safety, because unconditional evidence can be helpful or harmful. The paper builds a draft-conditioned selective retrieval framework and compares it with closed-book and always-retrieval settings. Selective Retrieval preserves closed-book behavior for low-need cases while controlling degradation associated with unconditional retrieval.

  • Problem

    The paper asks which single-turn mental-health queries should receive external evidence, since retrieval can improve specificity but also introduce noise or safety-sensitive guidance.

  • Method

    The paper fine-tunes one generator with QLoRA, builds a compact guideline corpus, and activates retrieval using draft-conditioned utility scores plus a rule-based safety trigger.

  • Results

    Always Retrieval improves specificity but lowers overall quality and adds safety-sensitive failures, whereas Selective Retrieval avoids the additional degradation caused by unconditional retrieval.

  • Takeaways & Limitations

    Retrieval activation should be treated as a safety-sensitive control decision in single-turn mental-health QA.

  • Takeaways & Limitations

    The study uses one open-source generator family, two splits from one benchmark framework, a 40-document corpus, and primarily LLM-based evaluation with a small expert audit.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) can improve the specificity and grounding of large language model responses, but its effect is not uniformly beneficial in single-turn mental-health question answering, where user queries often combine emotional distress, treatment concerns, and safety-sensitive needs. We study when retrieval helps or hurts mental-health QA, and whether a lightweight selective retrieval policy can better control this trade-off. We operationalize retrieval need using three draft-conditioned utility dimensions: psychoeducational need, coping need, and response specificity, together with a rule-based safety trigger. Following psychotherapy-grounded RAG systems such as coTherapist, we construct a compact and controllable guideline corpus comprising coping-strategy, psychoeducational, and safety resources. We fine-tune an instruction-tuned generator on MentalChat16K using QLoRA and compare Closed-book, Always Retrieval, and Selective Retrieval settings on CounselBench-Eval and CounselBench-Adv. Experiments show that retrieval is not uniformly beneficial in this domain. Always Retrieval improves specificity but lowers overall quality and introduces additional safety-sensitive failures. Selective Retrieval preserves closed-book behavior for low-need cases while avoiding the additional degradation caused by unconditional retrieval, supporting the view that retrieval activation is a safety-sensitive control decision.

1 Introduction

Single-turn mental-health QA requires responses that balance empathy, specificity, factual caution, and safety, making retrieval a selective control problem rather than an unconditional benefit. The paper proposes domain-specific retrieval activation based on support needs and safety risk.

  • Mental-health queries can combine distress, symptoms, treatment concerns, and coping requests, requiring empathetic and specific responses without unsafe guidance.
  • Retrieved evidence may be generic, weakly related, or overly directive, so Always Retrieval can improve specificity while introducing noise and safety risks.
  • Existing adaptive RAG criteria such as complexity and confidence do not fully capture needs for safety grounding, coping guidance, or psychoeducation.
  • The proposed policy treats retrieval as a domain-specific control problem using psychoeducation, coping support, response specificity, and a hard safety trigger.
  • The controlled comparison evaluates Closed-book, Always Retrieval, and Selective Retrieval with the same domain-adapted generator.

2 Related Work

Prior work applies RAG and adaptive retrieval to knowledge-intensive or psychotherapy-grounded settings, but this paper defines retrieval need around mental-health support functions and evaluates that policy in standard and adversarial QA.

  • Mental-health QA requires balancing empathy, specificity, factual caution, and professional boundaries beyond ordinary answer accuracy.
  • RAG combines parametric knowledge with external evidence, but noisy contexts can make retrieval less reliable than closed-book generation.
  • Adaptive retrieval methods commonly use query complexity, self-reflection, information need, utility criteria, or uncertainty.
  • This work instead defines retrieval need through psychoeducation, coping support, response specificity, and safety, evaluating the policy on standard and adversarial mental-health QA.
  • Psychotherapy-grounded systems such as coTherapist use authoritative therapy and clinical resources rather than unrestricted web evidence.

3 Preliminaries

The paper studies supportive response generation for single-turn mental-health QA using a domain-adapted generator with optional evidence retrieval from an authoritative guideline corpus.

  • The task is to generate a supportive response y for a user query q in single-turn mental-health QA.
  • The generator is domain-adapted on MentalChat16K, a benchmark of synthetic and anonymized counseling-related question-answer pairs.
  • At inference, the model may optionally retrieve supporting evidence from a small corpus of authoritative mental-health resources.

4 Methodology

The framework isolates retrieval-policy effects by holding a QLoRA-adapted generator fixed while comparing closed-book, always-retrieval, and draft-conditioned selective retrieval over a compact, source-typed guideline corpus.

  • 4.1 Fine-Tuning Base Generator: QLoRA adapts a single instruction-tuned generator on MentalChat16K, and that generator remains fixed across all retrieval conditions.
  • 4.1 Fine-Tuning Base Generator: Figure 1 organizes the framework into generator adaptation, BM25 indexing of a source-typed corpus, and inference-time selective retrieval.
  • 4.2 Guideline Corpus Construction: The guideline corpus follows psychotherapy-grounded retrieval practice and organizes evidence into coping, psychoeducational, and safety resources.
  • 4.2 Guideline Corpus Construction: The corpus contains 40 documents covering coping strategies, psychoeducation about common conditions, crisis guidance, urgent help-seeking, and medication caution.
  • 4.2 Guideline Corpus Construction: Documents are cleaned, segmented into overlapping word-level chunks, and stored with source-family and document metadata.
  • 4.2 Guideline Corpus Construction: BM25 retrieves top-k chunks from the corpus, with k=3 in the main experiments, keeping the setup focused on activation policy rather than retriever optimization.
  • 4.3 Inference-time Selective Retrieval: The three inference policies are Closed-book, Always Retrieval, and Selective Retrieval.
  • 4.3 Inference-time Selective Retrieval: Selective Retrieval first generates a closed-book draft, then scores the query and draft for information, coping, and specificity needs before deciding whether to retrieve.

5 Experiments

The experiments test retrieval policies under a fixed tuned generator and show that retrieval benefits depend on activation. Selective Retrieval limits safety-sensitive degradation while retaining useful behavior.

  • Experimental Design: The controlled comparison evaluates Closed-book, Always Retrieval, and Selective Retrieval under the same tuned generator.This isolates retrieval-policy effects from generator differences.
  • Main Results: Always Retrieval improves specificity over Tuned Closed-book but lowers overall quality and raises the medical-advice flag rate.The trade-off appears within the tuned variants rather than in comparison with the Base LM reference.
  • Main Results: Selective Retrieval improves Overall and Empathy over Tuned Closed-book while preserving a zero medical-advice rate.Its benefit is characterized as a quality-safety trade-off rather than a large average-score gain.
  • Safety Stress Test: Always Retrieval increases macro failure rate on CounselBench-Adv, especially through therapy- and assumption-related failures.CounselBench-Adv directly probes targeted unsafe or undesirable behaviors.
  • Safety Stress Test: Selective Retrieval matches the shared closed-book macro failure rate while activating retrieval for only 7.5% of adversarial questions.This limits the additional failures associated with unconditional retrieval under safety stress testing.
  • Expert Audit: The expert audit provides a targeted qualitative check rather than clinical validation of the automatic evaluation pattern.It focuses on professional boundaries, overly directive advice, and practical helpfulness.
  • Threshold Calibration: The selective policy uses a mean retrieval-need threshold and a high-axis route threshold to control activation.The mean threshold aggregates information, coping, and specificity needs, while the route threshold responds to strongly expressed informational or coping needs.
  • Threshold Calibration: Retrieval activation drops sharply at τ= 2.5 and remains near the hard-safety floor at larger thresholds; the main setting uses τ= 3.25.The sweep indicates that higher thresholds mainly preserve safety-triggered retrieval and a small number of high-need cases.

6 Limitations and Conclusion

The paper frames retrieval activation as a safety-sensitive control decision: unconditional retrieval can degrade safety, whereas selective retrieval limits that degradation. Its conclusions are bounded by the evaluated models, datasets, corpus, audit, and single-turn setting.

  • Conclusion: Always Retrieval improves specificity but can introduce additional safety-sensitive failures through overly directive, clinical, or medicalized guidance.Selective Retrieval instead preserves closed-book behavior for low-need cases and activates evidence under explicit utility or safety triggers.
  • Conclusion: Selective Retrieval’s central benefit is controlling degradation introduced by unconditional retrieval.The paper presents retrieval activation as a control problem rather than assuming retrieval is beneficial for every query.
  • Limitations: The gate combines fixed safety patterns with utility scores produced by the same generator used for response generation.Independently calibrated or learned policies may improve robustness.
  • Limitations: Validation uses one open-source generator family and two splits from the same benchmark framework, limiting claims of generality.The authors call for evaluation across model families and independently constructed mental-health datasets.
  • Limitations: The compact 40-document corpus improves source controllability but limits evidence coverage, while evaluation relies primarily on LLM judges and a small expert audit.These design choices constrain evidence breadth and the extent of human reliability checking.
  • Limitations: The single-turn setting does not capture longitudinal context, evolving retrieval needs, or multi-turn repair behavior.Future work includes temporally grounded retrieval for multi-session interactions.
  • Future Work: Future work extends evaluation to multiple generator families, independent datasets, larger evidence collections, learned gates, hybrid retrieval, and larger expert evaluations.These directions aim to separate retrieval-policy effects from model-family and dataset-composition effects.

A.3 Experimental Setup

The study adapts a generator on MentalChat16K and evaluates it on two CounselBench splits targeting general response quality and adversarial safety failures.

  • MentalChat16K provides synthetic counseling QA pairs and anonymized intervention transcripts for generator adaptation.
  • CounselBench-Eval contains 100 real patient questions with clinically grounded response-quality dimensions.
  • CounselBench-Adv contains 120 expert-authored adversarial questions targeting six mental-health failure categories.
  • The MentalChat16K-adapted QLoRA adapter and implementation code are publicly available.

A.1 Threshold Ablation

The threshold ablation tests whether broader retrieval activation improves the quality-safety trade-off, finding that the conservative threshold remains preferable.

  • Threshold comparison: Lowering τ from 3.25 to 2.25 increases retrieval activation from 9.0% to 38.0% on CounselBench-Eval and from 7.5% to 43.3% on CounselBench-Adv.
  • Threshold comparison: On Eval, overall quality and specificity slightly decrease under the lower threshold.
  • Threshold comparison: On Adv, macro failure increases from 0.0250 to 0.0417, mainly because of assumption failures.
  • Conclusion: The results support τ=3.25 as the conservative operating point.

A.2 Implementation Details

The implementation uses a fixed domain-adapted generator, a compact guideline corpus, and a selective gate that determines when retrieval is activated and how outputs are evaluated.

  • Generation and routing: The pipeline fixes one MentalChat16K-adapted generator across Closed-book, Always Retrieval, and Selective Retrieval conditions.
  • Generation and routing: Selective Retrieval generates a closed-book draft, scores information, coping, and specificity needs from 1 to 5, then either returns the draft or regenerates with evidence.
  • Corpus and retrieval: BM25 retrieves top-k=3 evidence, while the corpus, retrieval depth, generation settings, and judging scripts remain fixed in threshold ablations.
  • Corpus and retrieval: The guideline corpus contains coping, psychoeducation, and safety resources segmented into overlapping chunks for indexing.
  • Evaluation: CounselBench-Eval reports Overall, Empathy, Specificity, and Medical Advice Yes Rate, whereas CounselBench-Adv reports six failure rates and their macro average.
  • Evaluation: A targeted expert audit provides a qualitative reliability check for whether automatic retrieval-related patterns are clinically plausible.
Loading 2609.03454v1…