Source-linked AI summary
QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
Belinda Z. Li, Been Kim, Zi Wang
TL;DR
LLMs often face reasoning queries missing information needed for a determinate answer, but their ability to acquire that information is not well measured. The paper formalizes clarification-question selection with 1-sufficient CSPs and introduces QuestBench, finding strong math performance but weaker logic and planning performance. The results show that solving fully specified reasoning problems is insufficient for selecting the right question.
Problem
Existing reasoning benchmarks typically assume complete information, while real-world queries can omit details needed to solve the task.
Method
The paper formalizes underspecified reasoning as CSPs with one missing sufficient variable and evaluates clarification-question selection across four QuestBench task types.
Results
Models exceed 80% accuracy on GSM-Q and GSME-Q but remain below 50% on Logic-Q and Planning-Q.
Takeaways & Limitations
Success on QuestBench requires more than reasoning with sufficient information, indicating that information acquisition is a distinct capability.
Takeaways & Limitations
QuestBench’s design choices limit benchmark generalizability, and the framework currently focuses on one-missing-variable cases rather than general k-sufficient CSPs.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown impressive performance on reasoning benchmarks like math and logic. While many works have largely assumed well-defined tasks, real-world queries are often underspecified and only solvable by acquiring missing information. We formalize this information-gathering problem as a constraint satisfaction problem (CSP) with missing variable assignments. Using a special case where only one necessary variable assignment is missing, we can evaluate an LLM's ability to identify the minimal necessary question to ask. We present QuestBench, a set of underspecified reasoning tasks solvable by asking at most one question, which includes: (1) Logic-Q: logical reasoning tasks with one missing proposition, (2) Planning-Q: PDDL planning problems with partially-observed initial states, (3) GSM-Q: human-annotated grade school math problems with one unknown variable, and (4) GSME-Q: equation-based version of GSM-Q. The LLM must select the correct clarification question from multiple options. While current models excel at GSM-Q and GSME-Q, they achieve only 40-50% accuracy on Logic-Q and Planning-Q. Analysis shows that the ability to solve well-specified reasoning problems is not sufficient for success on our benchmark: models struggle to identify the right question even when they can solve the fully specified version. This highlights the need for specifically optimizing models' information acquisition capabilities.
1. Introduction
QuestBench studies whether LLMs can identify and ask for missing information in underspecified reasoning tasks. It formalizes this challenge with 1-sufficient CSPs and finds strong math performance but substantially weaker logic and planning performance.
- Real-world reasoning queries may omit crucial details, requiring LLMs to proactively gather missing information through clarification questions.
- QuestBench formalizes information gathering as an underspecified CSP in which the target variable cannot be inferred from the available assignments and constraints.
- 1-sufficient CSPs require acquiring one unknown variable value sufficient to compute the target, enabling evaluation through a single clarification-question choice.
- QuestBench covers Logic-Q, Planning-Q, GSM-Q, and GSME-Q, spanning logical reasoning, partially observed PDDL planning, and verbal or equation-based grade-school math.
- Over 80% accuracy on GSM-Q and GSME-Q contrasts with under 50% on Logic-Q and Planning-Q across evaluated state-of-the-art LLMs.
- Analysis across difficulty axes and ablations shows that QuestBench success requires more than reasoning with sufficient information alone.
2. Related work
Prior information-gathering benchmarks often emphasize ambiguity, subjectivity, or user simulation. QuestBench instead targets objectively determinable clarification questions for underspecified reasoning tasks.
- Information gathering is studied across concept learning, active learning, Bayesian optimization, reinforcement learning, and robot planning under partial observability.
- Natural-language requests can be ambiguous because of semantics, factual questions, dialogue intents, or personalized preferences.
- This paper distinguishes underspecification from ambiguity: underspecification means insufficient information to fulfill a request, while ambiguity permits multiple interpretations.
- Most existing information-gathering benchmarks address subjective or ambiguous tasks where multiple clarifying questions may be valid.
- Prior LLM question-asking methods mainly address ambiguous or knowledge-based tasks and may require substantial user simulation.
- QuestBench introduces objectively determinable clarification questions for underspecified reasoning tasks.
3. Problem formulation
The paper models reasoning queries as CSPs and defines underspecification through whether assignments suffice to determine a target variable. QuestBench restricts evaluation to 1-sufficient cases, whose difficulty can be characterized by search structure and variable or constraint counts.
- 3.1. Distinguishing semantic ambiguity and underspecification: A user request is parsed into variables, constraints, and a target variable before the model solves for that target in the resulting CSP.
- 3.1. Distinguishing semantic ambiguity and underspecification: The CSP formulation separates underspecification, caused by missing assignments or constraints, from semantic ambiguity, caused by multiple natural-language interpretations.
- 3.2. Formalizing underspecification: A CSP consists of variables X, domains D, constraints C, assignments A, and target variable y.
- 3.2. Formalizing underspecification: A variable is known when its value is directly assigned or derivable from existing assignments and constraints; a sufficient set makes the target known.
- 3.2. Formalizing underspecification: The smallest sufficient set identifies the minimal information needed for efficient question asking, and a 1-sufficient CSP has a smallest sufficient set of size 1.
- 3.3. Why 1-sufficient CSPs?: QuestBench accepts any sufficient variable when multiple equally minimal choices exist and uses 1-sufficient cases as a tractable basis for future k-sufficient extensions.
- 3.3. Why 1-sufficient CSPs?: Difficulty is described by the numbers of variables and constraints, backward-search depth, and expected brute-force guesses.
- 3.4. Difficulty axes of 1-sufficient CSPs: In the egg example, |X| = 3, |C| = 1, d = 1, and the expected brute-force guesses equal 2 because only x_0 is sufficient.
4. Dataset descriptions for QuestBench
QuestBench constructs multiple-choice question-asking instances across logical reasoning, planning, and math domains. Each instance supplies candidate clarification questions and evaluates whether the model selects a correct one.
- Each QuestBench instance contains a user request, all question choices, and a subset of correct questions for multiple-choice evaluation.
- 4.1. Logic-Q: Logic-Q adapts propositional logic problems involving rules, known properties of Alice, and an unknown target property.
- 4.1. Logic-Q: For Logic-Q, properties are Boolean variables, known facts form assignments, rules form implicative constraints, and the queried property is the target variable.
- 4.1. Logic-Q: Logic-Q constructs new assignments so one additional property is sufficient to determine whether the target property is true or false.
Logic-Q
The framework represents reasoning tasks as constraint satisfaction problems, with variables, domains, known assignments, action-dynamics constraints, and a target shortest action sequence. Planning-Q includes partially observed initial states where at most one additional atom may be needed to disambiguate the shortest path.
- Logic-Q example: The example figure places the language-model prompt on the left and the symbolic CSP used to construct questions on the right.The ground-truth answer is highlighted in red.
- CSP formulation: The CSP represents atoms as variables whose Boolean assignments describe the initial state.Each atom is assigned True or False, while A records atoms already known to be true or false.
- CSP formulation: Blocks World actions are governed by constraints linking each action’s preconditions and effects to consecutive states.The constraints require every current-state and next-state pair to satisfy the selected action’s preconditions and effects.
- Planning target: The target variable y is the shortest action sequence from the initial state to the goal state.Planning-Q uses this target to determine whether uncertainty about the initial state changes the required plan.
- Question construction: Planning-Q mixes 1-sufficient and well-specified problems, with at most one unknown atom needed to disambiguate the shortest action sequence.For well-specified instances, the shortest action sequence is invariant to the initial-state uncertainty, so no question is needed.
5. Results and analyses
QuestBench evaluates clarification-question accuracy across reasoning domains and relates performance to problem complexity and reasoning ability. Models perform strongly on math variants but struggle on Logic-Q and Planning-Q, and solving well-specified problems does not ensure effective question selection.
- Evaluation: Models were evaluated in zero-shot, chain-of-thought, and four-shot settings, with representative subsets used for GSM-Q and GSME-Q.The evaluation included closed and open models, with 288 GSM-Q and 151 GSME-Q tasks selected to focus resources on harder domains.
- Evaluation: QuestBench measures whether models select a correct clarification question from multiple choices, with breadth-first search as a reference baseline.Each benchmark item has ground-truth question choices, and BFS is evaluated to a fixed depth for comparison.
- Results: Neither chain-of-thought nor few-shot prompting produced significant gains across all models on Logic-Q or Planning-Q.Models generally remained at or below 50% in both domains, while math-domain performance was generally saturated.
- Complexity analysis: In Logic-Q, accuracy correlated negatively with backwards-search depth, variable count, constraint count, and expected brute-force guesses.These correlations were statistically significant for most models, suggesting that both search procedures approximate the reasoning required in this domain.
- Complexity analysis: In Planning-Q, accuracy showed weak or no correlation with backwards-search depth, variable count, or branching factor, while some models correlated with expected brute-force guesses.This pattern suggests mechanisms unlike backwards search and possibly behavior resembling choice-by-choice verification or guessing.
- Ablations: Filtering for tasks whose well-specified versions models could solve left Logic-Q question-asking effectively unchanged and Planning-Q only slightly improved.Neither domain reached 100%, indicating that solving the underlying specified problem does not guarantee identifying the missing information.
- Ablations: GSM-Q and GSME-Q question-asking accuracy changed by as much as +39% for some models and about −10% for others after filtering.The mixed changes indicate that well-specified reasoning ability is relevant but not sufficient for selecting the needed question.
6. Discussion and Conclusion
QuestBench evaluates whether LLMs can identify underspecification and ask clarification questions in reasoning problems. The benchmark exposes weaknesses in complex logic and planning tasks and highlights limitations and open directions for information-gathering evaluation.
- QuestBench introduces a benchmark for identifying underspecification and asking clarification questions in reasoning problems.
- SoTA LLMs perform relatively well on missing-information detection in simple algebra but struggle on complex logic and planning tasks.
- Asking the right question requires more than reasoning with sufficient information.
- Limitations: QuestBench’s design decisions improve evaluation ease and rigor but limit benchmark generalizability.
- Open problems: Future work includes k-sufficient CSPs, realistic user simulation, and metrics beyond clarification-question correctness.
- Evaluation format: The multiple-choice format is positioned as a rigorous precursor to open-ended question-asking evaluation.
B.1. Logic-Q
The Logic-Q construction derives assignments that imply a target proposition or its negation, then removes one variable assignment to create a 1-sufficient question-asking problem. Additional checks ensure the withheld variable is necessary and sufficient.
- Logic-Q construction begins by discarding assignments already present in the source dataset.
- Recursive backwards search derives assignment sets that imply the target proposition is true or false.
- Candidate 1-sufficient assignments are created by removing one variable assignment from paired assignments supporting opposite target values.
- Further checks reject candidates when the remaining assignment already determines the target or when alternative resolving assignments exist.
- The search tracks disjunctions of conjunctions and expands them through constraints that imply the target.
- Planning-Q: Planning-Q analogously withholds literals from partially observed initial states and tests whether one additional atom disambiguates the optimal plan.
B.3. GSME-Q
GSME-Q converts human-annotated math word problems into variables, equations, assignments, and a goal variable, while screening for ambiguity and translation errors. Its construction treats missing variables or equations as clarification targets.
- Annotators first solve each word problem and discard cases that are semantically ambiguous or do not match the source answer.
- Three annotators translate solvable problems into variables, equations, assignments, and a goal variable, followed by automated correctness checks.
- Ambiguity screening: The construction exposes ambiguities involving references, initial values, percentages, timing, and unspecified scoring details.
- The annotation interface asks for a solution before requesting clarification questions or an equation-based representation.
- Equation representation: The canonical basket example represents the second basket as B = 2 * A and the total as T = A + B, with T as the goal.
- Annotation rules: Annotators are instructed to record directly stated variable values but not values that can only be computed.
- Construction complexity: The backwards-search construction has runtime bounded by |X||C|^d when search depth is limited to d.
E. Qualitative Study of QuestBench Reasoning Traces
The qualitative study includes reasoning traces from Gemini 2.5 Pro on Logic-Q and Planning-Q problems because the Gemini 2.0 Flash Thinking Experimental model used originally was no longer available.
- Gemini 2.5 Pro traces are included because the originally evaluated Gemini 2.0 Flash Thinking Experimental model was no longer available.
E.1. Logic-Q
The Logic-Q example tests whether a model can ask for a missing proposition that determines a target statement’s truth. The model reasons extensively but selects an insufficient question because it investigates only how to prove the target true.
- Task setup: The task asks the model to determine whether Alice is tired by asking at most one permitted attribute question.The model must output a question only when the available facts and rules do not determine the target statement.
- Model reasoning: The model derives rational, dull, silly, and powerful from the initial facts before pursuing deductions about whether Alice is tired.Its trace uses both forward and backward chaining and repeatedly applies rules involving the target attribute.
- Model reasoning: The model prioritizes asking whether Alice is precious as the quickest route to a conclusion.It frames the choice as an information-efficient question while respecting the prohibition on asking about sensible.
- Outcome: The model answers incorrectly: sufficient questions are whether Alice is different, frightened, or hurt.The trace suggests it exhaustively explores positive derivations but does not adequately analyze routes to proving that Alice is not tired.
- Outcome: Precious is sufficient only when true, whereas the correct questions determine the target regardless of whether their answers are positive or negative.This distinction explains why the model’s selected question cannot resolve the target statement in both cases.
E.2. Planning-Q
The Planning-Q example presents a partially observed blocks-world state and asks which single observation can disambiguate a plan to the goal. The model correctly identifies whether block c is clear as sufficient resolving information.
- Problem instance: The goal requires placing c on b, b on a, and a on the table, starting from a partially specified arrangement of five blocks.The example includes the initial relations and a target tower that must be constructed through legal block actions.
- Task setup: Planning-Q gives a PDDL blocks-world problem with known facts, a goal state, and possible questions about predicates in the initial state.The model must select one question or indicate that no question is needed.
- Disambiguation: The key uncertainty concerns whether d is on the table or c is clear, because either fact determines the other.The model analyzes both possible initial-state scenarios and treats these variables as linked.
- Plan construction: The example’s plan strategy is to dismantle the existing tower, place blocks on the table, and reassemble them into the goal tower.The proposed actions must accommodate either possible initial configuration while respecting the empty-hand constraint.
- Outcome: The model correctly asks whether (clear c) is true as a minimal piece of information resolving the planning ambiguity.The analysis reports that this question is sufficient and connects the result to the model’s visualization-based reasoning.
G. Limitations
QuestBench prioritizes formal evaluation of one-question information acquisition, but its assumptions constrain generalizability to more complex and naturalistic settings.
- Scope assumptions: QuestBench evaluates only 1-sufficient CSPs, where one missing variable can resolve the target, while real problems may contain multiple missing pieces or complex dependencies.The paper describes this as a minimal information-acquisition case chosen for formalism and ease of evaluation.
- Scope assumptions: The selected domains have clear CSP formalizations but may not represent the full range of natural-language understanding and everyday ambiguity.This limits how directly benchmark results transfer to naturalistic tasks.
- Evaluation design: Multiple-choice question selection simplifies evaluation and may overestimate performance relative to open-ended question generation.The benchmark therefore does not fully test the difficulty of generating clarifying questions freely.
- Implications: The authors position the benchmark as a first attempt that favors formalism and evaluation ease over generality.They also connect improved information seeking to clarifying underspecified inputs in high-stakes domains.