PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoning

Yongqi Tong, Zhenyu Zhang, Zimi Liu, Kewei Fu, Mingli Song, Haofei Zhang, Junshao Zhang, Hong Zhu, Jiang-Ming Yang, Xin Zhang, Jianshe Li

arXiv:2608.16554v1cs.CL

TL;DR

Many reasoning queries omit premises needed for unique answers, requiring models to ask, condition responses, or abstain rather than guess. ACA-RL trains these behaviors and improves missing-premise robustness across model families while preserving competitive well-posed reasoning performance.

  • Problem

    Reasoning models lack evidence on handling queries whose omitted premises prevent a unique answer without defaulting to confident solutions or unhelpful refusal.

  • Method

    ACA-RL generates missing-premise instances and trains a structured behavioral policy over hallucination, assumption, abstention, conditional formulation, and active elicitation.

  • Results

    Across multiple model families, ACA-RL improves missing-premise and unanswerable benchmark behavior while preserving competitive well-posed reasoning performance; Qwen3-8B reaches 51.73 MPB Behavior Score versus 8.66 for Vanilla PPO.

  • Takeaways & Limitations

    The work frames reasoning progress as recognizing underdetermination and responding constructively, not only answering fully specified questions.

  • Takeaways & Limitations

    Active Elicitation is evaluated as a single-turn response, so ACA-RL does not train retrieval, tool use, or multi-turn clarification to acquire missing premises.

Abstract

from arXiv · show

Answer-only reinforcement learning (RL) trains reasoning models to solve fully specified problems, but many realistic queries omit a premise needed for a unique answer. In this setting, the useful response is not always refusal: the model should ask for the missing premise, condition its answer on the unknown quantity, or abstain when no informative conditional response is available. We present \emph{Ask-Condition-Abstain Reinforcement Learning} (ACA-RL), a data-augmented RL framework for this setting. Its reasoning-graph-guided pipeline converts well-posed problems into missing-premise training instances with localized gap annotations; ACA-RL then trains on these instances with a structured reward over five observable response behaviors. We also introduce the \emph{Missing-Premise Benchmark} (MPB), a 274-instance human-verified benchmark spanning mathematical, logical, and real-world word problems. Across Qwen3 and Llama models, ACA-RL consistently improves on MPB while preserving competitive performance on well-posed reasoning tasks. Together with the released code, MPB, and training data, this work supports a new mission for NLP evaluation: measuring whether models can recognize when a task is underdetermined and handle uncertainty, not only whether they can answer fully specified questions.

1 Introduction

The paper extends answer-only reinforcement learning to underdetermined prompts, where models should ask for missing premises, condition responses on them, or abstain. ACA-RL generates localized missing-premise training instances, optimizes five response behaviors, and evaluates them with the human-verified MPB benchmark.

  • ACA-RL: 120K natural missing-premise instances are generated by decomposing well-posed problems, perturbing critical premises, and adding localized gap annotations.The reasoning-graph-guided pipeline supplies ACA-RL’s data augmentation.
  • ACA-RL: ACA-RL optimizes a structured reward over five behaviors: silent hallucination, explicit assumption, abstention, conditional formulation, and active elicitation.The framework learns the missing-premise response policy within the model rather than relying on retrieval or an extra inference-time screening step.
  • MPB: 274 human-verified instances comprise MPB, spanning six perturbation types across mathematical, logical, and real-world word problems.MPB maps responses to the same five-category taxonomy and reports an averaged Behavior Score.
  • Results: ACA-RL improves MPB and third-party unanswerable-benchmark scores over Vanilla PPO, IDK-RL, and LM Introspection while remaining competitive on well-posed reasoning.The reported gains are not reduced to generic refusal: abstention appears early, whereas conditional formulation and active elicitation improve more gradually.
  • Motivation: Missing-premise reasoning is formulated as an answer-only RL extension in which useful responses ask, condition, or abstain.The work frames reasoning progress as recognizing when a task is underdetermined and handling the missing premise appropriately.

2 Preliminary

Missing-premise problems remove at least one premise required for a unique answer, so useful models must avoid fabricating information and instead abstain, condition their answers, or ask for the gap. MPB evaluates these terminal behaviors across 274 human-verified instances and six perturbation types.

  • Missing-premise problems: A missing-premise problem is a well-posed reasoning task perturbed by removing at least one premise needed for a unique answer.The perturbed instance includes a localized gap annotation.
  • Response behaviors: Useful responses should not silently fabricate missing information; they should abstain, formulate the answer conditionally, or ask for the missing premise.Ask and condition behaviors are preferred to generic refusal, while unsupported definite answers are penalized.
  • Evaluation protocol: 274 human-verified instances across six perturbation types comprise MPB’s behavioral evaluation protocol for whether models expose, parameterize, or request missing information.The protocol evaluates terminal response behavior rather than probability calibration and distinguishes silent hallucination, explicit assumption, abstention, conditional formulation, and active elicitation.

3 Missing-Premise Data Construction

The paper constructs missing-premise data by surgically perturbing well-posed reasoning problems, preserving their reasoning structure while localizing a single informational gap. A three-stage synthesis pipeline rewrites and filters these instances for scalable, uncertainty-aware training.

  • Dataset construction: 120K generated instances form the training set, while MPB uses a separate held-out candidate pool for evaluation.The construction treats missing-premise data as a controlled resource rather than arbitrary synthetic corruption.
  • Reasoning-graph-guided synthesis: The pipeline parses well-posed problems into components and builds a directed acyclic reasoning graph exposing dependencies between conditions and answers.The graph guides identification of critical conditions on the solution path.
  • Surgical condition perturbation: Critical conditions are perturbed to create underspecified problems with a single defined informational gap, paired with an annotation documenting the missing premise.The resulting pair is (s′, agap), preserving the original reasoning structure while altering a key constraint.
  • Rewrite and verification: The rewritten instances appear fully specified but are filtered for reasoning-graph correctness and missing-premise unanswerability before inclusion.An LLM-based quality filter discards samples that fail either check.
  • Rewrite and verification: 93.0% accuracy, 94.4% precision, 91.4% recall, and 92.9% F1 are achieved by the filter on 556 human-annotated held-out candidates.These annotations validate the filter’s unanswerability assessment for scalable training-data construction.

4 Ask-Condition-Abstain RL

ACA-RL trains models to handle missing premises through useful uncertainty responses rather than simple refusal or probability calibration. Its structured behavioral reward distinguishes hallucination, assumptions, abstention, conditional answers, and clarification requests, prioritizing responses that expose or resolve the missing information.

  • Behavioral objective: ACA-RL targets useful missing-premise handling by rewarding models that ask for the missing premise, condition answers on unknown variables, or abstain when neither is useful.The evaluation focuses on avoiding hallucination and communicating the missing premise usefully, rather than estimating calibrated probabilities.
  • Reward hierarchy: The reward hierarchy assigns 1.0 to Active Elicitation, 0.6 to Conditional Formulation, 0.3 to Abstention, −0.3 to Explicit Assumption, and −1.0 to Silent Hallucination.Positive rewards favor acceptable behaviors, while unsupported assumptions and silent hallucinations receive negative rewards.
  • Reward design: The structured reward replaces binary answer-or-refusal supervision with a fine-grained categorical signal that guides policies away from unsupported answers.The reward function is designed to encourage explicit engagement with informational gaps.
  • Behavior taxonomy: The behavior classifier partitions response trajectories into Silent Hallucination, Explicit Assumption, Abstention, Conditional Formulation, and Active Elicitation.These categories distinguish fabricated definite answers, stated assumptions, refusal, variable-based formulas, and proactive clarification questions.
  • ACA-RL objective: Optimizing J_ACA shifts probability mass away from hallucination toward conditional formulation and active elicitation on underspecified problems.The objective provides training supervision for navigating uncertainty rather than reproducing fully specified answer paths.

5 Experiments: Missing-Premise Behavior and General Reasoning

ACA-RL improves missing-premise behavior across model groups while preserving competitive performance on well-posed reasoning benchmarks. Experiments use a held-out 274-instance MPB and GPT-5-based categorical Behavior Scores.

  • Experimental Scope: The experiments also analyze data source, mixture ratio, training steps, and data size to characterize Behavior Score improvements and trade-offs with standard reasoning.The section covers missing-premise robustness, third-party unanswerable benchmarks, and well-posed reasoning checks.
  • Evaluation Protocol: The evaluation uses 120K ACA-RL training instances and a separate 274-instance MPB selected from a held-out, validity-verified candidate pool.GPT-5 judges responses by mapping them to Section 4 behavior categories, whose arithmetic means define Behavior Scores.
  • Missing-Premise Behavior: ACA-RL obtains an MPB Behavior Score of 51.73 on Qwen3-8B, versus 8.66 for Vanilla PPO and 48.72 for IDK-RL.The comparison shows stronger missing-premise robustness than answer-only RL and a smaller gap relative to refusal-trained IDK-RL.
  • General Reasoning: On LiveBench, ACA-RL matches the Vanilla PPO/RLVR average at 59.0 versus 59.0, while SciBench scores are 53.7 versus 56.0.These results test whether missing-premise training causes over-refusal on well-posed reasoning tasks.

6 Analysis: Data and Behavior Trade-offs

The analysis shows that reasoning-graph-guided data, balanced missing-premise mixtures, and cold-start SFT improve informative uncertainty handling while preserving standard reasoning. ACA-RL shifts behavior beyond generic refusal through ask/condition responses, benefits from continued training and larger datasets, and outperforms prompting alone on MPB.

  • Data Source for Missing-Premise Behavior: Reasoning-graph-guided synthesis achieves the highest MPB Behavior Score while maintaining competitive GSM8K and AIME’24 performance under a fixed data budget.This indicates that the synthesis procedure contributes beyond simply adding unanswerable examples.
  • Portion of Missing-Premise Questions: 50% missing-premise data yields the highest MPB Behavior Score but lower general reasoning, whereas 10% preserves standard reasoning with weaker missing-premise behavior.The 30% default is used as a practical balance and preserves substantially more general reasoning than 50% while improving MPB over 10%.
  • From Abstention to Ask/Condition Behavior: ACA-RL attains higher Behavior Scores than IDK-RL while retaining slightly lower but competitive IDK scores, shifting responses from generic refusal toward informative ask/condition behavior.Behavior Score rewards conditional formulation and active elicitation in addition to refusal.
  • Cold-start SFT: Cold-start SFT accelerates ACA-RL learning on MPB and improves its final score under the same training budget, while RL further refines the ask/condition/abstain policy.Without initialization, learning is slower and the final MPB score is lower.
  • Training Steps: Additional training steps sustain MPB growth while general reasoning stabilizes, and conditional formulation and elicitation gradually displace some IDK responses.Figure 3 reports relatively stable general benchmark scores, while Figure 4 shows early abstention followed by more gradual conditional formulation and elicitation.
  • Comparison with Training-Free Methods: 51.73 vs. 9.58: ACA-RL obtains a higher MPB score than the training-free verbalized-uncertainty prompting baseline.The comparison suggests explicit missing-premise training is more effective in this setting than prompting alone, without ruling out stronger prompting or screen-then-answer systems.
  • Training Data Size: Larger generated datasets improve later-stage GSM8K and MPB performance across the 10k, 20k, and 52k sample sweep.The experiment supports scalable missing-premise data construction but does not establish a full scaling law.

7 Related Work

Prior work improves reasoning, studies missing or unanswerable questions, and explores clarification and uncertainty communication. ACA-RL complements these directions by training a hierarchy of ask, condition, and abstain responses for missing-premise inputs while penalizing unsupported answers.

  • Reinforcement Learning for Reasoning: RL-based reasoning methods optimize outcome or process rewards mainly for fully specified tasks, whereas ACA-RL targets missing, ambiguous, or contradictory premises.
  • Missing-Premise and Unanswerable Questions: Existing benchmarks construct unanswerable or underspecified questions to evaluate hallucination and abstention, including human-annotated and dependency-edge-removal approaches.
  • Clarification, Ambiguous QA, and Uncertainty-Aware Reasoning: Clarification and ambiguous-QA research studies asking users to resolve ambiguity or recovering multiple interpretations, while ACA-RL makes asking one trained response among several.
  • Abstention and Verbalized Uncertainty: ACA-RL extends abstention and verbalized-uncertainty methods with a rewarded hierarchy—ask, condition, then abstain—and penalties for unsupported assumptions and hallucinated answers.

8 Conclusion and Future Work

The work proposes a learned response policy for underdetermined questions, training models to ask, condition, or abstain rather than fabricate answers. ACA-RL, together with MPB, forms a training and evaluation framework for missing-premise behavior.

  • Conclusion and Future Work: ACA-RL trains on missing-premise problems generated by a reasoning-graph pipeline.The pipeline supports training on questions whose premises do not determine a unique answer.
  • Conclusion and Future Work: Structured behavioral rewards encourage models to ask, condition, or abstain instead of fabricating a value.These behaviors constitute the learned response policy for underdetermined questions.
  • Conclusion and Future Work: ACA-RL and MPB provide a concrete framework for training and evaluating missing-premise behavior.The framework is intended for questions whose premises do not determine a unique answer.

Limitations … E Experiment Details

The paper identifies limitations in ACA-RL’s single-turn, synthetic-data setting and behavior-based evaluation, while documenting MPB construction, diagnostics, reasoning-graph generation, and behavioral scoring examples.

  • Limitations: ACA-RL identifies missing premises but does not train retrieval, tool use, or multi-turn clarification policies to acquire them in agentic workflows.This leaves a gap between recognizing underspecification and resolving it.
  • Limitations: Synthetic perturbations of logical problems may miss the messy, implicit, and semantic ambiguity of organic user queries.Robustness is therefore primarily verified against logical incompleteness rather than the full spectrum of open-world ambiguity.
  • Limitations: Training rewards and MPB scoring use the same behavior taxonomy, so reported margins over IDK-RL reflect gains under the Behavior Score rubric.The paper calls for independent scoring protocols and larger-scale human scoring.
  • A MPB: Construction and Diagnostics: MPB addresses an evaluation gap by measuring behavior on underspecified or inconsistent problems through a hierarchy distinguishing five response behaviors.The categories are silent hallucination, explicit assumption, abstention, conditional formulation, and active elicitation.
  • A MPB: Construction and Diagnostics: No listed model is uniformly strong across MPB’s six perturbation categories, supporting MPB’s use as a diagnostic benchmark.The benchmark-level Behavior Score aggregates ask, condition, and abstain behavior across the categories.
  • B Condition Perturbation Definitions and Examples: MPB instances are constructed with condition-perturbation operations that remove enough information for direct answers while permitting confident but potentially incorrect responses.Detailed definitions and examples are provided in Table 8.
  • C Reasoning-Graph Example: The reasoning-DAG example shows original constraints progressively combined into intermediate inferences that yield a complete arrangement and final answer.The example is shown in Figure 7.
  • D Behavioral Scoring Example: A behavioral-scoring example contrasts silent hallucination with a preferred conditional formulation when parabolic equations and constraints are insufficient to determine a+b.The undesired response assumes a symmetric configuration and unstated equations before returning a numeric value.

E.1 Training and Evaluation Details … E.4 Additional Evaluator Agreement Check

The paper evaluates ACA-RL across Qwen3 and Llama models using missing-premise, out-of-distribution, and well-posed reasoning benchmarks, with behavior-based MPB scoring and binary IDK scoring. Human annotations largely agree with GPT judging, supporting the consistency of the MPB evaluation protocol.

  • E.1 Training and Evaluation Details: ACA-RL is evaluated with Qwen3-8B, Qwen3-14B, and Llama3.1-8B-Instruct models, using Qwen3 cold-start SFT checkpoints for subsequent RL.The Qwen3 models represent reasoning-heavy architectures, while Llama3.1-8B-Instruct represents an instruction-tuned model.
  • E.1 Training and Evaluation Details: Evaluation covers MPB, UMWP, and SUM for unanswerable-question robustness, plus AIME’24, MATH-500, and GSM8K for general reasoning.The missing-premise training data is synthesized exclusively from DeepscaleR, with answerable questions taken directly from its original dataset.
  • E.1 Training and Evaluation Details: Reported metrics are Behavior Score and IDK score for MPB, UMWP, and SUM, and Pass@1 for GSM8K and MATH-500 plus average Pass@1 over 8 samples for AIME’24.MPB uses Behavior Score and IDK score, while general reasoning benchmarks use Pass@1-based measures.
  • E.2 MPB Scoring: MPB responses are classified into five behavior categories, mapped to discrete Behavior Scores, and averaged arithmetically across test instances.This produces continuous benchmark-level values despite discrete scores for individual responses.
  • E.3 Binary IDK Scoring: IDK scoring uses a binary rubric assigning 0 to incorrect and 1 to correct responses, with an appended instruction requesting boxed “I don't know” answers when uncertain.The protocol follows Song et al.’s evaluation approach.
  • E.4 Additional Evaluator Agreement Check: MPB scoring uses GPT-5 to assign each response to one terminal behavior category and map it to the Table 9 reward or Behavior Score.Three graduate-degree human experts independently annotated sampled GPT-5 and Qwen3-235B-A22B-Instruct responses using the same labels.
  • E.4 Additional Evaluator Agreement Check: Approximately 98% of sampled GPT-5 responses and 95% of sampled Qwen3-235B-A22B-Instruct responses received human labels agreeing with the GPT judge.The agreement check supports consistency of the MPB scoring protocol and evaluator independence.

F Data Synthesis Details … G Use of LLMs

The data-synthesis pipeline uses LLM-agent stages to decompose problems, generate reasoning graphs, surgically remove necessary premises, reconstruct fluent instances, and verify correctness and missing-premise unanswerability. LLMs also support data filtering and behavioral judging, while authors retain responsibility for the research design, analysis, and claims.

  • F Data Synthesis Details: The released supplement will provide prompt templates for decomposition, graph generation, perturbation, reconstruction, and rechecking to support reproducibility.The paper summarizes these stages rather than embedding raw prompt files in the main text.
  • F.1 Problem Decomposition: Problem decomposition separates source problems into background information, explicit conditions, and the target query for graph construction.This representation supplies the input structure for reasoning-graph generation.
  • F.2 Reasoning Graph Generation: Reasoning-graph generation converts each solution into a directed acyclic graph with conditions or intermediate conclusions as nodes and dependencies as edges.The graph represents the structure used to guide subsequent premise perturbation.
  • F.3 Surgical Condition Perturbation: Surgical perturbation selects a condition on the reasoning path and removes or alters a premise necessary for a unique answer while preserving fluency and plausibility.The operation is chosen from the condition-breaking operations listed in Table 8.
  • F.4 Reconstruction: Reconstruction rewrites the perturbed conditions, original background, and target query into a fluent problem statement.This stage turns the edited components into a complete missing-premise instance.
  • F.5.1 Reasoning Correctness: Reasoning-correctness checking verifies that the original reasoning graph and answer remain coherent before perturbation.This recheck protects the source problem’s reasoning validity before the premise is removed or altered.
  • F.5.2 Missing-Premise Unanswerability: Missing-premise checking verifies that the perturbed problem lacks enough information to determine the original answer and that the induced gap is identifiable.The check targets both unanswerability and localization of the missing premise.
  • G Use of LLMs: LLMs support data synthesis, filtering, and GPT-based behavioral judging, while authors remain responsible for research questions, method design, analysis, and final claims.General-purpose writing assistants may separately provide minor wording, grammar, and formatting support.
Loading 2608.16554v1…