PaperlayerView exact revision on arXiv

Source-linked AI summary

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Skylar Zhai, Jingcheng Liang, Dongyeop Kang

arXiv:2604.17073v1cs.CLcs.AI

TL;DR

The paper addresses the problem that reinforcement fine-tuning can make models guess on unanswerable queries, while existing methods often use generic refusals or unvalidated clarifications. It introduces clarification-aware RLVR rewards and trains Abstain-R1 to jointly abstain and identify missing information. Across the reported benchmarks, Abstain-R1 improves unanswerable-query calibration and clarification while preserving strong answerable-query performance.

  • Problem

    Reinforcement-learning objectives can encourage hallucinated answers to semantically clear but unresolvable queries, while existing abstention methods may not verify clarification quality.

  • Method

    The paper trains a 3B model with clarification-aware RLVR that rewards strict abstention and semantically aligned post-refusal clarification while retaining rewards for correct answerable-query responses.

  • Results

    ABSTAIN-R1 improves refusal calibration and clarification quality on unanswerable queries while preserving strong performance on answerable ones across ABSTAIN-TEST, ABSTAIN-QA, and SELFAWARE.

  • Takeaways & Limitations

    Reliable abstention with useful clarification can be learned through dedicated optimization with verifiable rewards rather than emerging automatically from scale or standard post-training.

  • Takeaways & Limitations

    The evaluation focuses mainly on English QA-style benchmarks, and clarification rewards and evaluation rely on LLM-based judges that may introduce bias or miss valid clarifications.

Abstract

from arXiv · show

Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.

1 Introduction

The paper addresses reliable behavior on semantically clear queries that cannot be resolved from available information, where reinforcement learning can encourage confident guessing. It proposes verifiable rewards for explicit abstention and actionable clarification, and evaluates these behaviors alongside answerable-query performance.

  • Motivation: Reinforcement learning can encourage confident guesses on unanswerable queries, making fluent hallucinations especially risky in high-stakes domains.The paper frames explicit non-answering as safer than inventing missing conditions or premises.
  • Motivation: Unanswerable queries are semantically clear but lack a uniquely solvable or reliably inferable answer from the provided information.Examples include missing or underconstrained conditions, false premises, internal contradictions, and objectively unavailable answers.
  • Research gap: Existing approaches teach refusals through supervised fine-tuning or coarse reinforcement-learning objectives, but may produce brittle, generic, or unvalidated clarification behavior.The paper distinguishes generic refusal from clarification that identifies the information preventing resolution.
  • Approach: The proposed clarification-aware RLVR reward jointly rewards strict abstention and semantically aligned post-refusal clarification while preserving performance on answerable queries.Rule-based checks verify the abstention format, while an additional reward measures semantic alignment with a reference clarification.
  • Evaluation: The evaluation measures appropriate binary abstention and finer-grained clarification quality, including clarification consistency and actionability.The reported metric suite also captures performance retention on answerable queries.
  • Contributions: ABSTAIN-R1 is a 3B model trained to improve abstention calibration and clarification quality while maintaining performance on answerable queries.The contribution is paired with ABSTAIN-TEST, a metric suite for evaluating abstention and post-refusal clarification.

2 Related Work

Prior work documents failures to abstain and limitations of generic refusal strategies, while reasoning-focused reinforcement learning usually assumes queries are solvable. This paper positions calibrated refusal with validated post-refusal clarification as a distinct target.

  • Abstention and calibration: Mainstream language models often fail to abstain appropriately on unanswerable questions across diverse settings.Related work also reports high-confidence invention of missing constraints when necessary conditions are omitted.
  • Abstention and calibration: Some abstention approaches improve calibration and answered-subset accuracy beyond knowledge boundaries, but can reduce unconditional accuracy.The paper distinguishes these efforts from its focus on underspecified queries and clarification quality.
  • Positioning: The paper’s related-work distinction is calibrated refusal under underspecified queries with explicit evaluation of post-refusal clarification quality.This contrasts with generic refusal patterns and coarse abstention behavior.
  • Existing limitations: Supervision-centric refusal gains can become brittle outside curated distributions, while other methods encourage follow-up questions without validating their content.This motivates evaluating whether post-refusal clarification identifies the information needed for resolution.
  • Reasoning-focused RL: Reasoning-focused reinforcement-learning methods use structured or verifiable rewards for complex tasks but generally optimize correctness while assuming query solvability.The paper identifies explicit refusal rewards for unanswerable scenarios as missing from most such methods.

3 Dataset

The paper builds supervised and evaluation datasets around semantically clear but unresolved queries, then adds clarification-focused reinforcement-learning data. These resources span multiple domains and pair abstention with actionable identification of missing information.

  • ABSTAIN-COT: ABSTAIN-COT is a supervised fine-tuning dataset designed to test whether explicit abstention and clarification behaviors affect later reinforcement-learning training.It is built from ABSTENTIONBENCH and follows the paper’s definition of unanswerable queries.
  • ABSTAIN-COT: ABSTAIN-COT targets outputs that explicitly abstain and then ask an actionable clarification question or identify the key missing information.Samples contain a reasoning trace in <thinking> and a final response in <answer>.
  • ABSTAIN-COT: 4.6K ABSTAIN-COT samples cover mathematics, life sciences, reading comprehension, fact-checking, world knowledge, ethics, social bias, and medical reasoning.The dataset is generated from original questions using rule-based and domain-specific prompting.
  • ABSTAIN-TEST: ABSTAIN-TEST uses the same selected task subsets and generation pipeline as ABSTAIN-COT, additionally incorporating the SUM test set for targeted clarification evaluation.The resulting evaluation set contains approximately 2.9K instances.
  • RL data: Reinforcement learning uses the SUM training split without overlap with its evaluation split to provide clarification-style supervision signals.The training split contains 50K paired instances.
  • RL data: Mixed reinforcement-learning sampling uses roughly 30% unanswerable and 70% answerable queries to encourage targeted clarification while maintaining answerable-query performance.The sampling scheme jointly exposes the policy to abstention and ordinary answering behavior.

4 Method

The method combines supervised initialization with GRPO-based RLVR, using verifiable rewards to optimize answerability-sensitive behavior. It rewards correct answers on solvable queries and explicit, clarification-supported abstention on unsolvable ones.

  • Training pipeline: Abstain-SFT initializes the policy on curated Abstain-CoT data to strengthen instruction adherence, output formatting, and clarification reasoning.Reasoning traces cover both answerable and unanswerable questions.
  • Training pipeline: GRPO samples groups of outputs and uses their relative composite rewards to guide policy updates without a separate value model.The optimization remains constrained by a frozen reference model and KL regularization.
  • Reward design: The composite reward combines format adherence, answer correctness, abstention logic, and clarification quality.This design jointly targets answerable and unanswerable-query behavior.
  • Reward design: Strict output formatting requires tagged reasoning and answers, with boxed answers for solvable queries and boxed “I don’t know” for unsolvable ones.The format reward supports stable parsing of the reasoning and final result.
  • Reward design: For answerable queries, symbolic verification rewards ground-truth matches and penalizes refusals on solvable problems.The answerable reward is intended to mitigate under-confidence.
  • Reward design: For unanswerable queries, explicit boxed abstention earns partial credit, while a verifier-confirmed clarification identifying missing information earns full reward.The refusal-with-clarification reward assigns 0.3 for abstention and an additional 0.7 for a correct clarification.

5 Experiments

The evaluation measures answer retention, abstention, and clarification quality across three benchmarks and a diverse model pool. Clarification metrics are reported only where the benchmark supplies the required annotations.

  • Evaluation metrics: Six metrics cover answerable accuracy, false refusals, conditional answer accuracy, unanswerable refusal, clarification, and conditional clarification rates.A-Acc and A-FU assess answerable queries, while U-Ref, U-Clar, and U-Clarc assess unanswerable behavior; A-Accc measures answered-query accuracy.
  • Evaluation metrics: Clarification quality is evaluated by rewriting each question into a meta-level query and comparing model and reference clarifications as explanations of unanswerability.The framework uses semantic equivalence rather than requiring identical wording.
  • Evaluation setup: RL training uses the conservative 3B xVerify-3B-Ia verifier, while offline evaluation uses the stronger o4-mini with the same rewrite and parsing rules.The two verifiers serve training and offline evaluation, respectively.
  • Evaluation setup: Experiments span Abstain-Test, Abstain-QA, and SelfAware, covering open-source models at multiple scales, proprietary systems, and Qwen2.5 3B variants.The model pool includes Qwen2.5 3B/7B/32B, Llama3.1 8B, DeepSeek-V3, and DeepSeek-R1.
  • Evaluation setup: Abstain-Test reports all six metrics, whereas Abstain-QA reports A-Acc, A-FU, and U-Ref, and SelfAware reports U-Ref.The latter benchmarks lack annotations needed for post-refusal clarification metrics; Abstain-QA is adjusted to align with the unanswerability protocol.

6 Results and Analysis

The experiments evaluate Abstain-R1 across answerable and unanswerable queries, training dynamics, component ablations, reward design, and simpler alternatives. Results show improved abstention and clarification while preserving answerable performance, with reward choices shaping the trade-off.

  • RQ1: Abstain-R1 achieves the strongest overall behavior on unanswerable queries among evaluated models, including refusal correctness, clarification quality, and consistency.Its unanswerable-query performance remains competitive with or stronger than substantially larger off-the-shelf models.
  • RQ1: Abstain-R1 improves refusal behavior on unseen ABSTAIN-QA and SELFAWARE inputs, attaining the strongest refusal performance on SELFAWARE.Larger instruction-tuned or RL-tuned models do not show monotonic gains in abstention reliability.
  • RQ2: Abstain-R1 improves answerable-question accuracy across benchmarks with only a modest increase in false refusals.On ABSTAIN-TEST, conditional answer accuracy also increases, indicating higher correctness among questions the model chooses to answer.
  • RQ3: During RL training, response length first rises slightly and then decreases steadily, while abstention rate, clarification correctness, and answer accuracy all improve.The gains are larger for abstention and clarification than for answer accuracy.
  • RQ4: SFT provides an initial foundation, while RL further improves refusal and clarification; unanswerable training data is essential for learning abstention.Removing the clarification reward mainly reduces clarification quality while leaving refusal relatively strong.
  • RQ5: Answerable-side penalties reduce over-abstention and recover answerable performance, whereas stronger clarification rewards do not improve all metrics monotonically.The best balance appears at an intermediate clarification-reward value, and reward design directly determines the answerability–reliability trade-off.
  • RQ6: SFT improves more strongly and stably than ICL, while ABSTAIN-R1 achieves a better overall trade-off than SFT-ALL without external chain-of-thought distillation.SFT-ALL has stronger unanswerable-query performance but worse answerable performance and false-refusal rate.

7 Conclusion

The paper presents Abstain-R1, a 3B model trained with clarification-aware RLVR to preserve correct answering while improving abstention and post-refusal clarification. Across three benchmarks, it improves unanswerable-query behavior and supports clarification as a dedicated training and evaluation target.

  • Conclusion: Abstain-R1 uses clarification-aware RLVR to preserve correct answers while improving abstention and post-refusal clarification on semantically clear but unresolved queries.The objective explicitly rewards both abstention and clarification correctness.
  • Conclusion: Across ABSTAIN-TEST, ABSTAIN-QA, and SELFAWARE, Abstain-R1 improves refusal calibration and clarification quality while preserving strong answerable-query performance.The results support dedicated optimization with verifiable rewards rather than relying on scale or standard post-training alone.
  • Conclusion: Post-refusal clarification is presented as an important target for future training and evaluation in broader settings.The paper identifies multilingual, open-ended, and tool-augmented environments as directions for future work.

Limitations

The paper’s evaluation and training setup has several scope, measurement, safety, and deployment limitations. Transfer beyond English QA-style benchmarks and the reliability of LLM-based judges remain uncertain.

  • Scope: Evaluation focuses mainly on English QA-style benchmarks, leaving transfer to open-ended, multilingual, or tool-augmented settings unclear.This limits conclusions about broader deployment contexts.
  • Measurement: Training rewards and clarification evaluation rely on LLM-based judges, which may introduce bias and miss valid clarification forms.Judge-based evaluation may not capture the full diversity of acceptable clarifications.
  • Scope: The study targets unanswerability and underspecification, while other hallucination forms and safety risks remain outside scope.The conclusions therefore concern a defined subset of reliability problems.
  • Deployment: RLVR training adds computational cost and requires careful tuning of verifiers and reward scales, limiting direct production practicality.These requirements may complicate deployment of the training setup.

A Implementation Details

The implementation fine-tunes Qwen2.5-3B-Instruct with full-parameter SFT and then applies GRPO-based reinforcement fine-tuning. Training uses four A100 GPUs, with reported hyperparameters provided for reproducibility.

  • SFT: The SFT stage fine-tunes Qwen2.5-3B-Instruct on ABSTAIN-COT with full-parameter updates and selects the best checkpoint from 10 epochs.The selected checkpoint is Epoch 3.
  • RL: The reinforcement stage uses Group Relative Policy Optimization within the PPO framework for reinforcement fine-tuning on the SUM training dataset.GRPO avoids a separate value model and reduces memory requirements.
  • Compute: Training is conducted on a single node with four A100 GPUs, and 100 Qwen2.5-3B-Instruct training steps require roughly 20 A100 GPU hours.The reported hardware and cost characterize the implementation scale.
  • Reproducibility: Tables 5 and 6 summarize the hyperparameters used in the SFT and RL stages to facilitate reproducibility.The hyperparameter summaries cover both major training stages.

B Dataset Processing

The evaluation pipeline combines diverse answerable and unanswerable datasets with SFT and GRPO training, then validates clarification judgments across domains. The verifier agrees substantially with o4-mini but is conservative on SUM, making SFT initialization important.

  • Training: The training pipeline uses SFT initialization followed by GRPO reinforcement fine-tuning, with key configurations documented for both stages.
  • Dataset construction: The evaluation suite covers diverse unanswerable-query subsets spanning Alcuna, BBQ, FalseQA, GSM8K-Abstain, Known-Unknown-Questions, MediQ, Moral-Choice, Musique, QAQA, SQuAD2, UMWP, and World-Sense.
  • Dataset construction: Abstain-Test follows the CoT construction pipeline, uses a smaller UMWP quota, and adds SUM paired questions with higher-quality clarification supervision.
  • Dataset construction: SelfAware evaluates refusal on unanswerable questions alongside semantically similar answerable questions, covering multiple sources of unanswerability.
  • Verifier validation: 92.9% agreement is observed in Medical, 87.0% in Biology, 80.5% in Reading Comprehension, and 79.6% in World Knowledge between the verifier and o4-mini.
  • Verifier validation: Among 174 SUM clarifications, the verifier has 2 false positives relative to o4-mini but 94 cases where it rejects clarifications o4-mini accepts.

C.2 Per-Domain Results on ABSTAIN-TEST and ABSTAIN-QA

Abstain-R1 improves abstention and clarification across ABSTAIN-QA and ABSTAIN-TEST while generally preserving answerable accuracy. The strongest gains occur in aligned or safety-sensitive domains, though conservative refusal can modestly reduce accuracy.

  • ABSTAIN-QA: On MMLU, Abstain-R1 maintains strong A-Acc while raising U-Ref above the nearly always-answering DeepSeek-R1 regime.
  • ABSTAIN-QA: On CQA and PopQA, Abstain-R1 improves U-Ref beyond the 3B baseline while keeping A-Acc essentially unchanged, indicating transfer to long-tail knowledge domains.
  • ABSTAIN-TEST: Across eight ABSTAIN-TEST domains, Abstain-R1 raises both U-Ref and U-Clar over Qwen2.5 3B Instruct while keeping A-Acc comparable or slightly improved.
  • ABSTAIN-TEST: In Math, Abstain-R1 improves refusal accuracy, clarification clarity, answerable performance, and false-refusal control where reward-model alignment is strongest.
  • ABSTAIN-TEST: In Biology, Medical, and Ethics, Abstain-R1 shifts the 3B model toward more frequent refusals and clearer explanations, with modest A-Acc decreases in some domains.
  • ABSTAIN-TEST: In fact-checking, reading comprehension, and world knowledge, RL training increases refusal frequency and quality, with domain-dependent changes in A-Acc and A-FU.

D Qualitative Case Studies of Calibrated Abstention

Case studies show Abstain-R1 replacing unsupported answer production with explicit refusals that identify missing evidence across factual, medical, mathematical, and ethics scenarios. Its clarification evaluation uses a verifier-based prompt and shows strong but imperfect human alignment.

  • Case studies: Across four risk-sensitive domains, Abstain-R1 converts implicit uncertainty into explicit refusals while baseline systems continue answering or repairing unanswerable questions.
  • Fact-checking: In fact-checking, Abstain-R1 detects the contradiction in the apple-tree question instead of repairing it into an answer about apples.
  • Medical: In medical reasoning, Abstain-R1 flags omitted clinical details and abstains because multiple etiologies remain plausible.
  • Mathematics: In mathematics, Abstain-R1 identifies missing varnish quantities and refuses to compute a drying-time difference unsupported by the problem.
  • Bias/Ethics: In bias and ethics, Abstain-R1 rejects a comparison lacking evidence about parental involvement rather than relying on socioeconomic stereotypes.
  • Clarification evaluation: The clarification verifier reformulates an unanswerable question as a meta-question asking for the main reason it cannot be reliably answered, then compares the clarification with a reference.
  • Human evaluation: 86% agreement between o4-mini and consensus human labels indicates strong but imperfect alignment, with o4-mini often judging borderline useful clarifications too strictly.
Loading 2604.17073v1…