PaperlayerView exact revision on arXiv

Source-linked AI summary

MASH: Modeling Abstention via Selective Help-Seeking

Mustafa Omer Gul, Claire Cardie, Tanya Goyal

arXiv:2510.01152v2cs.CL

TL;DR

LLMs may not reliably recognize when questions exceed their parametric knowledge, while abstention-oriented alignment is described as reducing errors without expanding the set of answerable questions. MASH trains selective help-seeking with reinforcement learning and search penalties, then uses search requests as an abstention signal when search is unavailable. Across knowledge-intensive datasets, it outperforms prior efficient search baselines and shows strong abstention behavior.

  • Problem

    The paper asks how to train LLMs to recognize knowledge boundaries while also expanding the set of questions they can answer, beyond approaches that reduce errors without changing answerability.

    Source: §1 · p. 1

  • Method

    MASH uses reinforcement learning to reward answer accuracy while penalizing searches, treating help-seeking as a proxy for abstention when search access is removed.

    Source: §1 · p. 1

  • Results

    MASH outperforms prior efficient search baselines across three knowledge-intensive datasets; on multi-hop datasets, answer accuracy improves by 7.6%.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    MASH training yields a model that recognizes its boundaries while expanding its answerable question set through help-seeking.

    Source: §1 · p. 2

    For 2WikiMultiHopQA, the authors omit comparison and bridge-comparison questions from training and evaluation because their answer distributions may enable reward hacking.

    Source: §3 · p. 4

Abstract

from arXiv · show

LLMs cannot reliably recognize their parametric knowledge boundaries and often hallucinate answers to outside-of-boundary questions. In this paper, we introduce MASH (Modeling Abstention via Selective Help-seeking), a training framework that readily extracts abstentions from LLMs. Our key idea is that any external help-seeking by an LLM, i.e. search tool use, can serve as a proxy for abstention if the external help (search) is appropriately penalized while also rewarding answer accuracy. MASH operationalizes this idea using reinforcement learning with a pay-per-search reward. We run experiments on three knowledge-intensive QA datasets. Our results show that MASH substantially improves upon the selective help-seeking performance of prior efficient search approaches; on multi-hop datasets, it improves answer accuracy by 7.6%. Furthermore, MASH demonstrates strong off-the-shelf abstention performance, showcasing behavior competitive with prior abstention methods that additionally require predetermining model knowledge boundaries to construct training data. Overall, we show MASH training effectively aligns search tool use with parametric knowledge, which can be successfully leveraged for making abstention decisions and efficient search tool use

1. Introduction

MASH trains LLMs to seek help selectively as a route to recognizing knowledge boundaries while expanding what they can answer. Its results report stronger help-seeking performance and competitive off-the-shelf abstention.

1. Introduction

  • The proposed training strategy aims both to recognize knowledge boundaries and to expand the set of questions answerable with the model’s capabilities.Prior boundary-learning strategies are described as reducing errors without changing how many questions the model can correctly answer.Source: §1 · p. 1
  • MASH trains LLMs to seek help only when they cannot effectively answer a query alone, using selective help-seeking as an indirect route to abstention.The framework draws inspiration from people seeking outside help when they lack knowledge.Source: §1 · p. 1
  • MASH uses reinforcement learning with a pay-per-search reward and adds synthetic-data curation plus a lightweight SFT pipeline to improve exploration without boundary information.The reward discounts correctness according to searches; harsher penalties were found crucial for learning help-seeking behaviors.Source: §1 · pp. 1–2
  • On multi-hop datasets, MASH improves answer accuracy by 7.6% while producing a better distribution of searches.Its performance is on par with search baselines that allow searches without a penalty.Source: §1 · p. 2
  • MASH achieves competitive off-the-shelf abstention performance with DPO, while reporting 10 −20% higher answer accuracy over non-abstained questions than prompting and supervised abstention methods.The comparison is attributed to better differentiation between answerable and unanswerable questions.Source: §1 · p. 2
  • MASH’s results characterize it as an abstention model that recognizes its boundaries while expanding its answerable question set through help-seeking.The paper presents boundary recognition and expanded answerability as concurrent capabilities.Source: §1 · p. 2

2. MASH: Modeling Abstention via Selective Help-seeking

MASH frames abstention as selective help-seeking: search behavior reflects whether parametric knowledge suffices, and removing search access turns search requests into abstentions. A boundary-agnostic warm start and search-sensitive reward formulation support this training approach.

2.1. Abstention Framework

  • The framework trains for correctness while penalizing searches; at inference, a search request signals abstention when search access is removed.The optimal policy answers parametrically when expected parametric accuracy exceeds the maximum reward attainable by seeking help.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2.1. INITIALIZING πθ W/ WARM-START SFT

  • Warm-start SFT uses constrained decoding to generate trajectories with varied search counts, improving RL initialization without using knowledge-boundary information.The synthetic data is intentionally not aligned with correctness or parametric knowledge; the model learns search behavior during RL.Source: §2.2.1 · p. 3

2.2.2. REWARD FORMULATION

  • The reward multiplies binary correctness by a search penalty, with penalty severity shaping help-seeking and OTC-Strict assigning an extremely strict penalty when a correct zero-search trajectory exists.The formulation compares a trajectory’s search count with the most efficient correct trajectory; OTC sets c to the maximum allowed searches.Source: §2.2.2 · pp. 3–4

3. Experimental Setup

Experiments use three knowledge-intensive QA datasets with separately trained Qwen2.5-3B models, and assess help-seeking both with tools and abstention without them. The section specifies comparison baselines, metrics, and abstention evaluation.

3. Experimental Setup

  • Experiments use NaturalQA, HotPotQA, and 2Wiki, training Qwen2.5-3B separately on each with E5 retrieval over the 2018 Wikipedia dump.NaturalQA is single-hop; HotPotQA and 2Wiki are multi-hop, with different search strategies and distributions of parametrically answerable questions.Source: §3 · p. 4
  • Warm-start trajectories come from a separate Qwen2.5-32B base model, using 1,000 sampled training questions per dataset and five candidate samples per question.The separate base model avoids baking its knowledge boundaries or instruct-model abstention behavior into the SFT data.Source: §3 · p. 4
  • The 2Wiki comparison and bridge-comparison questions are omitted from training and evaluation because their skewed answer distributions could permit reward hacking.Source: §3 · p. 4

3.1. Details for Inference Mode I: w/ Search Tools Baselines

  • Search-enabled comparisons include parametric-only R1, unpenalized Search-R1, OTC, and three MASH reward variants; MASH w/ OTC also uses warm-start training.Source: Baselines · p. 4
  • Search-enabled evaluation reports judged answer accuracy, average tool calls, and tool productivity, which discounts each trajectory’s correctness by its search count.Checkpoint selection uses validation tool productivity except for Search-R1, which is selected by accuracy; metrics average four samples.Source: Baselines · p. 4

3.2. Details for Inference Mode II: Abstention

  • For abstention, the model loses search access at inference and is compared with prompting, honesty-alignment SFT, DPO, and ternary-reward RL baselines.Several baselines require answerability labels, estimated from ten samples using threshold η = 0.1.Source: §3.2 · pp. 4–5
  • Abstention evaluation measures overall accuracy, precision on answered questions, and abstention rates on always-incorrect versus always-correct questions; 2Wiki classification is excluded.Only 58 2Wiki test examples fall in the always-correct group, preventing reliable classification conclusions.Source: §3.2 · p. 5

4. Results

Across the reported evaluations, MASH balances answer accuracy and search use, shows competitive abstention, and generalizes across tested model scales. Warm-start stabilizes harsher penalties, whereas oracle helpers do not yield selective help-seeking.

4.1. Inference Mode I: w/ Search Tools

  • MASH w/ OTC-Strict improves tool productivity by 5.62 points over OTC on average and matches Search-R1 accuracy on HotPotQA and 2Wiki with fewer searches.On NaturalQA, OTC-Strict answers parametrically for 36% of questions with a 2.5% accuracy drop; Exp-Dec does so 35% of the time with a 4.5% accuracy drop and a 7-point productivity improvement.Source: §4.1 · pp. 5–6

4.2. Inference Mode II: w/ Abstention

  • MASH variants generally surpass prompting and Alignment for Honesty SFT on precision with comparable overall accuracy, while OTC-Strict is competitive with DPO across datasets.On HotPotQA, OTC-Strict has 59.9 precision versus DPO’s 53.1, but 17.3 overall accuracy versus 19.9; DPO’s higher abstention Delta can accompany lower precision.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3. Generalization to Other Model Scales and Families

  • Across Qwen2.5-7B and Qwen3-4B on HotPotQA, MASH w/ OTC-Strict continues to outperform baselines in search use and abstention.The study reports average improvements of 2.98% on accuracy and 9.59 on tool productivity; Qwen2.5-7B has higher precision than DPO (60.67% vs 48.25%), while Qwen3-4B has higher accuracy (20.96% vs 16.72%).Source: §4.3 · p. 7

4.4. Impact of Warm-start on Performance

  • Warm-start enables successful training runs for harsher penalties, and OTC-Strict with warm-start substantially outperforms OTC on abstention metrics across datasets.Without warm-start, harsher-penalty runs show severe instability, including policies collapsing to zero searches or collapsing on 2Wiki.Source: §4.4 · p. 7, Table 4 · p. 7

4.5. Do oracle helpers improve selective help-seeking?

  • Oracle-helper training converges to always seeking help within 50 steps, rather than selectively seeking help based on parametric knowledge.The paper attributes selective behavior with the retrieval helper partly to its noisiness and says better-exploring RL algorithms are needed for the oracle setting.Source: §4.5 · p. 8

4.6. Out-of-Distribution Performance

  • On out-of-distribution tests, MASH generalizes better than OTC, while HotPotQA-trained MASH with search achieves 26.43% higher accuracy than abstention-only DPO.For 2Wiki-trained models, MASH generalizes well to HotPotQA but performs more poorly on single-hop datasets.Source: §4.6 · p. 8, Table 5 · p. 8

5. Related Work

Related work places MASH among abstention training, selective retrieval, and tool-use methods. It contrasts prior boundary-estimation pipelines and specialized tool-use settings with learning boundaries through an efficient tool-use reward.

5. Related Work

  • Prior abstention pipelines estimate knowledge boundaries before constructing SFT or DPO data or training model-specific RLHF rewards.Other approaches train models to verbalize uncertainty through SFT data or structured multi-agent interactions.Source: §5 · p. 8
  • Unlike Collaborative Self-Play, MASH learns boundaries through efficient tool-use reward without multi-agent scaffolding or task-complexity assumptions.MASH also differs from OTC [Wang et al., 2025a], which analyzes efficiency without reference to parametric knowledge, by comparing reward formulations and abstention methods.Source: §5 · pp. 8–9
  • Selective RAG methods commonly estimate uncertainty through hidden states, sample consistency, or output probabilities before deciding when to search.The cited approaches include Yao et al. (2025) and Baek et al. (2025) for hidden states, Ding et al. (2024) for sample consistency, and Jiang et al. (2023) and Su et al. (2024) for output probabilities.Source: §5 · pp. 8–9

6. Conclusion

MASH trains selective help-seeking as a route to both search use and abstention behavior. The appendix’s theoretical setup simplifies the search objective and assumes parametric accuracy is imperfect for some questions.

6. Conclusion

  • MASH trains selective help-seeking and reports higher overall accuracy than prior efficient-search baselines when searches are allowed, alongside strong abstention behavior.The conclusion describes its abstention performance as analogous to specialized abstention methods.Source: §6 · p. 9

A.1. Preliminaries

  • The theoretical setup sets the KL penalty weight β to 0.0 and assumes parametric answering cannot achieve perfect expected accuracy for every question.The latter follows the paper’s training setup, in which RL does not add knowledge beyond the base model’s boundaries.Source: §A.1 · p. 13
  • The preliminaries define N_s(τ) as a trajectory’s search-call count and R*_i(q, y) as the maximum expected reward using i searches.Definition 2 introduces the accuracy/help reward relationship, though its full definition is not included here.Source: §A.1 · p. 13

A.2. Proof

The proof characterizes when an optimal policy answers parametrically rather than searching. The accompanying appendix details the GRPO, inference, and warm-start procedures used in the experiments.

A.2. Proof

  • The proof decomposes expected reward by search count, weighting each count’s conditional expected reward by its probability under the policy.This weighted-sum view supports comparing zero-search behavior with policies that search.Source: §A.2 · p. 13
  • An optimal policy answers parametrically if and only if maximum achievable parametric accuracy is at least the maximum achievable reward from searching.Lemma 1 expresses the comparison as no-search reward exceeding every positive-search reward; the theorem states the equivalent accuracy-versus-search-reward condition.Source: §A.2 · pp. 13–14

B. Search Tool Use

  • Appendix B describes GRPO training, warm-start construction, and the datasets used for training and evaluation.The section introduces implementation and data details rather than a new method claim.Source: B. Search Tool Use · p. 15

B.1. GRPO Training

  • GRPO training uses a 10^-6 learning rate without warmup or decay, runs for 400 steps, and evaluates on validation data every 25 steps.The implementation uses batch size 64, group size 16, and one gradient step per rollout.Source: §B.1 · p. 15

B.2. Inference Algorithm

  • Inference caps Search-R1 experiments at three searches for compute and memory reasons; efficiency-reward runs did not hit their maximum search limit.When a search limit is reached, the procedure returns a warning instead of performing another retrieval.Source: §B.2 · p. 15

B.3. Warm-Start

  • Warm-start data samples a target search count, builds think/search/answer trajectories, and selects a correct trajectory when available.For answer actions, candidate outputs are preferentially chosen when correct; the resulting trajectories are used for SFT.Source: §B.3 · pp. 15–16
  • Warm-start Delta is 1.55 on Natural Questions and 7.69 on HotPotQA, while Abs(0) is near 66%.The authors attribute the near-66% Abs(0) to two thirds of the warm-start data containing search behavior, and interpret the small Delta values as weak answerability alignment.Source: §B.3 · p. 16

B.4. Datasets

The dataset description compares question-answerability distributions across Natural Questions, HotPotQA, and 2WikiMultiHopQA. It reports that 2WikiMultiHopQA is more challenging than the other two datasets.

B.4. Datasets

  • Warm-start construction samples a search count, generates think/search/answer actions with retrieval, and selects a correct trajectory when one is available.The prompt figure for R1 training replaces its final question with the input question.Source: §B.4 · p. 17, Figure 2 · p. 17
  • 2WikiMultiHopQA has 77.26% of questions at average accuracy 0 and 0.85% at 1, compared with 44.72% and 11.26% for Natural Questions and 50.11% and 7.92% for HotPotQA.Natural Questions and HotPotQA have similar overall distributions, while 2WikiMultiHopQA is harder.Source: §B.4 · p. 17, Table 7 · p. 17

C. Abstention Experiment Details

This section describes how question answerability is estimated, how abstention baselines are trained and evaluated, and how a ternary-reward baseline behaves. The Ternary baseline collapses to always abstaining within 25 training steps.

C.1. Question Accuracy Estimation

  • Question accuracy is estimated from 10 sampled base-model responses, with few-shot examples and an LLM judge assessing extracted shortform answers.DeepSeek-V3.1 provides the five training-set examples and performs extraction and judging; Qwen2.5-3B generates the responses used to estimate its knowledge boundaries.Source: §C.1 · p. 18

C.2. Trained Abstention Model Details

  • Alignment for Honesty and DPO use MASH's training datapoints and gradient-step count; DPO uses SFT regularization instead of a two-stage SFT-then-DPO procedure.Alignment for Honesty uses SFT training; the DPO baseline sets β = 0.1 and its SFT loss coefficient to 1.Source: §C.2 · pp. 18–19
  • Both trained abstention baselines use the paper's abstention prompt and inference temperature of 1.0, without top-p or top-k sampling.Source: §C.2 · p. 19

C.3. Few-Shot Abstention Prompting Details

  • Few-shot abstention prompting uses four prompts with differently balanced answer and abstention examples for answerable and unanswerable questions.Each prompt contains five examples, and the answers are sampled from DeepSeek-V3.1.Source: §C.3 · p. 19

C.4. Ternary Abstention Baseline Details

  • Within 25 steps, the Ternary baseline converges to always abstaining.It uses GRPO rewards of +1 for correct answers, 0 for abstentions, and −1 for incorrect answers.Source: §C.4 · p. 19

C.5. Evaluating Abstention Models

  • During evaluation, responses are first checked for abstention; only non-abstaining responses proceed to answer extraction and correctness evaluation.Source: §C.5 · p. 20

D. Generalization to Other Model Scales and Families Details

The section tests MASH across model scales and families on HotPotQA, including its search-enabled performance and fine-grained search-use distribution. MASH w/ OTC-ST outperforms OTC on accuracy and tool productivity, with a 10% increase in tool productivity.

D. Generalization to Other Model Scales and Families Details

  • The scale and family experiments use Qwen2.5-7B-Base and Qwen3-4B-Base on HotPotQA, comparing OTC and MASH w/ OTC-Strict against abstention baselines except Ternary.The experiments are limited to 300 training steps rather than 400 because of compute limitations; Ternary is excluded because it learned to abstain on all questions for Qwen2.5-3B-Base.Source: D. Generalization to Other Model… · pp. 20–21
  • On HotPotQA, MASH w/ OTC-ST outperforms OTC on Accuracy and tool productivity (TP), with a 10% increase in TP.Source: Table 8 · p. 21
  • The fine-grained distribution reports TC=0, TC=1, and TC=2+ search groups and their answer accuracies; TC=0 indicates parametric answering.The caption reports that MASH off-loads questions from TC=1 to TC=0 with minimal or no accuracy decrease.Source: Table 9 · p. 21

E. Oracle Helper Details

With an oracle helper that returns the answer, MASH variants converge to always seeking help, an outcome the paper equates with abstention. The section relates this behavior to ternary-reward abstention and reports the associated prompt and metric setup.

E. Oracle Helper Details

  • Oracle-helper training replaces the search and document tags with help and helper-answer tags, and uses a fixed “I need help” message during warm-start generation.Source: E. Oracle Helper Details · p. 21
  • All MASH variants converge to one help request within 20 steps with an oracle helper, regardless of dataset, warm-start procedure, or penalty severity.The oracle-helper analysis describes this as always seeking help, which is equivalent to abstention under MASH.Source: E. Oracle Helper Details · pp. 21–22, Figure 11 · p. 21
  • An efficient oracle-helper policy needs at most one help call; correct answers, help-seeking, and incorrect answers therefore receive three reward levels.The paper identifies this as the same ternary reward structure used by its Ternary baseline, which collapsed to always abstaining.Source: E. Oracle Helper Details · pp. 21–22
  • The accompanying abstention table reports overall accuracy, non-abstained-answer precision, Abs(0), and the abstention-rate difference between unanswerable and answerable questions.Source: Table 10 · p. 22

F.1. Additional Abstention Metrics

MASH's abstention tendency tracks estimated question answerability, and MASH w/ OTC-Strict compares favorably with prompted Qwen2.5-3B-Instruct on the reported QA datasets. On separate general-capability tasks, the trained model shows no degradation and improves over the base model on both reported metrics.

F.1. Additional Abstention Metrics

  • Models' tendency to abstain decreases as a question's average accuracy increases; 2WikiMultiHopQA is omitted because most Abs(i) buckets lack sufficient support.Source: §F.1 · p. 22, Table 12 · p. 23

F.2. Instruct Model Prompting

  • MASH w/ OTC-Strict significantly outperforms zero- and few-shot Qwen2.5-3B-Instruct prompting in abstention evaluation, with higher answer accuracy while maintaining similar precision on Natural Questions.On HotPotQA and 2WikiMultiHopQA, the prompted instruct model abstains on most questions and has very low answer accuracy; MASH w/ OTC-Strict has substantially higher accuracy while maintaining precision.Source: §F.2 · p. 22, Table 13 · p. 23

F.3. Impact on General Task Performance

  • MASH w/ OTC-Strict improves over Qwen2.5-3B-Base by 1.60% on IFEval and 7.68% on MATH-Hard, with no observed degradation in general-task performance.Source: §F.3 · p. 23

G. Out-of-Distribution Results

Out-of-distribution results vary across datasets: the relative strengths of DPO, OTC, and MASH differ across abstention and accuracy measures, while tool use raises accuracy for OTC and MASH in the reported settings. The experiments required approximately 1900 H100 hours in total.

G. Out-of-Distribution Results

  • Selective-help-seeking training does not degrade general capabilities in the reported setting.The comparison uses Qwen2.5-3B-Base and HotPotQA-trained MASH w/ OTC-Strict on IFEval and MATH-Hard.Source: Table 14 · p. 24
  • On NaturalQA, DPO leads on Abs(0) and Delta, MASH is outperformed on TriviaQA, and OTC’s search behavior generalizes out-of-distribution.Tool use enables both OTC and MASH to achieve higher accuracies.Source: Table 15 · p. 24

H. Compute Requirements and Cost

  • The experiments used approximately 1900 H100 hours in total, and DeepSeek-V3.1 API calls cost approximately $400–500.The 18 full reinforcement-learning experiments account for approximately 1800 H100 hours.Source: H. Compute Requirements and Cost · p. 24
  • On HotPotQA, DPO leads on Abs(0) and Delta while MASH leads on Accuracy; OTC is more competitive with DPO than MASH on 2Wiki abstention metrics.Tool use enables both OTC and MASH to achieve higher accuracies.Source: Table 16 · p. 25
  • On single-hop datasets, DPO leads on Abs(0), OTC leads on Delta, and OTC and MASH lead on Accuracy; MASH struggles at abstention.Tool use enables both OTC and MASH to achieve higher accuracies.Source: Table 17 · p. 25

Related papers

Loading 2510.01152v2…