QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?

• The paper formalizes information gathering as an underspecified CSP, where the target cannot be inferred from given information, distinct from semantic ambiguity with solvable interpretations.Source: §1 · p. 2
• QuestBench tests 1-sufficient CSPs by asking models to select an unknown, non-derivable variable whose value suffices to compute the target.Source: §1 · p. 2
• The benchmark includes Logic-Q proposition tasks, Planning-Q partially observed PDDL blocks-world tasks, and human-annotated GSM-Q verbal and GSME-Q equation math problems.Source: §1 · p. 2
• Models exceed 80% accuracy on GSM-Q and GSME-Q but remain below 50% on Logic-Q and Planning-Q.Source: §1 · p. 2
• Difficulty analyses across variable and constraint counts, backward-search depth, and brute-force guesses show domain-dependent trends; ablations find sufficient-information reasoning alone does not ensure QuestBench success.Source: §1 · p. 2

Importance of Information Gathering

• Information gathering has been studied in human development and AI domains including active learning, Bayesian optimization, reinforcement learning, and partially observable robot planning; this work evaluates LLMs’ information gathering through basic reasoning tasks.Chouinard et al. (2007) connects question asking with children’s cognitive development; Wu (2023) emphasizes question asking by AI assistants facing uncertainty in code generation.Source: §2 · p. 3

Ambiguity in user requests

• Ambiguity research spans semantics, factual QA, dialogue intents, preferences, and text-to-image generation, while taxonomies distinguish uncertainty types and ambiguity categories (Kuhn et al., 2023b; Chandu et al., 2024; Zhang et al., 2024a).Chandu et al. (2024) studies epistemic and aleatory uncertainty, whose distinction can be unclear; Zhang et al. (2024a) includes unfamiliarity and semantic question types.Source: §2 · p. 3
• This paper focuses on underspecification—insufficient information to fulfill a request, potentially because users do not know what the model lacks or what is necessary—and evaluates it in structured reasoning tasks.Source: §2 · p. 3

Information gathering benchmarks for LLMs

• Many existing information-gathering benchmarks involve subjective or ambiguous tasks with multiple context-dependent clarifications, whereas this work uses reasoning tasks with ground truth and exactly one required question for reliable evaluation.Task-oriented dialogue and preference elicitation lack a universal right question (Budzianowski et al., 2018; Rastogi et al., 2020; Zhang et al., 2024b; Li et al., 2023).Source: §2 · p. 3

Question-asking methods for LLMs

• Existing LLM question-asking methods address ambiguous or knowledge-based tasks through direct prompting or information-gain ranking, while this work targets objectively determinable clarifications for underspecified reasoning.Zhang and Choi (2023) divide question asking into detecting need, identifying the question, and responding to new information; prior methods often use subjective tasks or substantial user simulation.Source: §2 · pp. 3–4

3.1. Distinguishing semantic ambiguity and underspecification

• The task separates parsing a query into a CSP from solving it, distinguishing missing CSP information (underspecification) from multiple possible query interpretations (semantic ambiguity).The paper focuses on underspecification, which it describes as objectively evaluable with its CSP framework.Source: §3.1 · p. 4

3.2. Formalizing underspecification

• A CSP comprises variables with domains, constraints, known assignments, and a target variable; a value is known if assigned directly or derivable from existing assignments and constraints.In the eggs example, knowing both the initial and eaten quantities determines the remaining quantity.Source: §3.2 · pp. 4–5
• A sufficient set is a set of variables whose known values determine the target, and efficient questioning seeks the smallest such set.For the eggs example, the initial quantity alone is sufficient to determine the target once the eaten quantity is known.Source: §3.2 · p. 5
• A 1-sufficient CSP has a smallest sufficient set of size one, and any sufficient variable is accepted when multiple such variables exist.QuestBench comprises 1-sufficient CSPs.Source: §3.2 · p. 5, §3.3 · p. 5

3.3. Why 1-sufficient CSPs?

• 1-sufficient CSPs provide an approximate upper bound on information-acquisition capacity and enable simple multiple-choice evaluation, unlike k-sufficient tasks that may require sequential questions.Any k-sufficient CSP can become 1-sufficient after k−1 clarifying questions.Source: §3.3 · p. 5

3.4. Difficulty axes of 1-sufficient CSPs

• 1-sufficient CSP difficulty varies and can be characterized by variable count, constraint count, backward-search depth, and expected random guesses to select a correct question.The described solution approaches are brute-force checking each variable and backward search from the target through constraints.Source: §3.3 · p. 5, §3.4 · pp. 5–6

4.1. Logic-Q

• Logic-Q replaces SimpleLogic4’s original assignments with new known facts so one added property determines whether the queried property is true or false.Its CSP represents Alice’s possible properties as Boolean variables, with known facts, implication rules, and a target property.Source: §4.1 · pp. 6–7

4.2. Planning-Q

• Planning-Q uses partially observed Blocks World initial states and shortest action sequences, mixing cases where up to one atom disambiguates the answer with cases needing no question.The CSP treats initial-state atoms as Boolean variables; its target is the shortest action sequence to the goal.Source: §4.2 · pp. 7–8

4.3. GSM-Q and GSME-Q

• Human annotators create GSM-Q and GSME-Q from GSM-Plus distractor problems, withholding irrelevant distractor assignments and one assignment necessary and sufficient to derive the target variable.GSME-Q retains the CSP form, while GSM-Q converts it back to a word problem with the missing assignment and distractors removed.Source: §4.3 · p. 9

5.1. Evaluation of clarification question accuracy

• Best models saturated GSME-Q and GSM-Q but did not perform beyond 50% on Logic-Q and Planning-Q; CoT and few-shot prompting brought no significant gains across models in either difficult domain.Source: §5.1 · p. 9, Table 2 · p. 10
• Models generally found verbalized GSM-Q harder than raw-equation GSME-Q, with symbolic forms hypothesized to make the search tree for identifying missing information easier to build.Source: §5.1 · p. 10

5.2. Correlation between problem complexity and clarification question accuracy

• In Logic-Q, accuracy correlated negatively with backward-search depth, variable and constraint counts, and expected brute-force guesses, with statistically significant correlations for most models.In Planning-Q, correlations with backward-search depth, variable count, and branching factor were very weak or absent; some models instead correlated with expected brute-force guesses.Source: §5.2 · pp. 11–13

5.3. Is asking the right question harder than solving the problem?

• On filtered examples with at least one correctly solved well-specified variant, Logic-Q question-asking accuracy stayed effectively unchanged and below 100%, Planning-Q rose slightly, and some GSM-Q/GSME-Q results shifted up to 39% or down about 10%.The filtered evaluation isolated question-asking on examples where the model could solve a corresponding well-specified task.Source: §5.3 · p. 13, Table 5 · p. 14, Table 6 · p. 15

5.4. Detecting underspecification and recognizing uncertainty

• Planning-Q “not sure” F1 was at most 5% better than Random for every model; zero-shot Gemini Flash Thinking, Claude 3.5, and GPT-4o flagged 1.4%, 4.6%, and 0.7% of cases versus 41.8% ground truth.The authors hypothesize that models may guess rather than recognize uncertainty on sufficiently difficult Planning-Q problems.Source: §5.4 · p. 14

6. Discussion and Conclusion

• QuestBench findings show that state-of-the-art LLMs handle missing information in simple algebra better than complex logic and planning, and that question-asking requires more than reasoning with sufficient information.Performance can be negatively correlated with complexity factors such as search depth and expected brute-force guesses.Source: §6 · p. 15
• QuestBench’s design decisions limit its generalizability, and the framework is a first step toward formalizing underspecification.The discussion presents open problems for future work.Source: §6 · p. 15
• The framework could extend to k-sufficient CSPs by holding out k variable values, with questions selected in a batch, sequentially, or batch-by-batch.Evaluation must specify whether all k questions are selected simultaneously or in stages.Source: §6 · pp. 15–16
• Table 7 reports that most LLMs struggle to identify underspecified problems, particularly in Planning-Q.The table includes a random baseline that labels a proportional number of examples “not sure.”Source: Table 7 · p. 16
• Automating realistic users who answer clarification questions is a bottleneck for evaluating iterative dialogue systems for complex tasks (Gui and Toubia, 2023).Such systems can require multiple clarification questions, including in medical diagnosis and intent elicitation.Source: §6 · p. 16
• Question-asking evaluation could measure more than correctness, including how easy questions are for users to answer (Grand et al., 2024; Hahn et al., 2025; Li et al., 2023).Source: §6 · p. 16
• The multiple-choice format supports automated, rigorous evaluation as a precursor to open-ended question-asking evaluation; the benchmark can later omit the options.The authors say this is possible once open-ended evaluation becomes viable and reliable.Source: §6 · p. 16

A. Author contributions

• All authors contributed to brainstorming and writing; Belinda Z. Li led dataset construction, evaluation, data collection, experiments, and empirical analysis, while Been Kim advised and Zi Wang proposed and initiated the project.Kim advised project direction and dataset and analysis work; Wang built prototypes, analyzed results, and advised dataset construction and human data collection.Source: A. Author contributions · p. 21

B. Dataset Construction Details

• The dataset construction derives assignments implying either goal value, then removes one variable assignment to create candidate 1-sufficient cases.The process starts from SimpleLogic problems after discarding assignments already present in the dataset.Source: §B.1 · p. 21, §B.1.1 · p. 23, §B.1.2 · p. 23

B.1. Logic-Q

• For Logic-Q, the construction discards assignments already in SimpleLogic and finds assignments that imply either y or ¬y.The two sets are obtained through recursive backwards search from y and ¬y.Source: §B.1 · p. 21, §B.1.1 · p. 23
• Candidate underspecified cases are formed by comparing assignments that imply opposite goal values and removing the assignment to one variable x_d.The construction requires x_d’s value to determine whether y or ¬y is true.Source: §B.1 · pp. 21–22

B.1.1. Constructing all assignments that imply goal is true

• Backwards search begins with y true and tracks a disjunction of conjunctions of variable assignments required for the goal to be true.It expands those conjunctions using rules that imply y.Source: §B.1.1 · p. 22
• The search recursively expands assignments within each conjunction, applies the distributive property, and repeats from ¬y to obtain assignments implying either goal value.This yields the full sets A(y) and A(¬y).Source: §B.1.1 · pp. 22–23

B.1.2. Checking 1-sufficiency

• Further checks discard candidates that fail the stated constraints, including candidates where x_d by itself is sufficient to determine y.The checks are applied after constructing potential 1-sufficient assignments.Source: §B.1.2 · p. 23
• To require search to the full depth needed for resolution, the construction creates an invalid set from resolution sets of alternative assignments and bars questions about specified variables.At test time, the LM is told it cannot ask about values in the sufficiency set of the candidate assignment.Source: §B.1.2 · p. 23

B.2. Planning-Q

• The Planning-Q setup models Blocks World states, actions, initial conditions, and manually specified goals, using breadth-first search to find shortest plans.An optimal action sequence is the shortest sequence that reaches a state satisfying the goal.Source: §B.2 · pp. 24–25
• Withholding literals from the initial state creates partial observations; well-specified partial states are those whose consistent full states share one optimal action sequence to the goal.The construction then removes one atom and checks whether asking for its truth value resolves the trajectory.Source: §B.2 · p. 25

B.2.1. Deriving all well-specified partial states with the same optimal action sequence to goal

• Starting from the goal conditions, the B.2.1 procedure searches backward breadth-first to derive optimal partial-state trajectories ending at the goal.It iteratively finds actions and partial states that could lead into trajectories found in the prior iteration.Source: §B.2.1 · pp. 25–26
• The search expands branches until a partial state already encountered implies the current one, then unions the resulting trajectories’ initial states to obtain the well-specified set.In the worst case, the search exhausts all possible partial states.Source: §B.2.1 · p. 26

B.2.2. Creating 1-sufficient partial states

• For each well-specified initial state, B.2.2 removes one proposition and accepts the resulting partial state only if one additional atom’s truth value can disambiguate its possible plans.The procedure checks that no more than two distinct optimal action sequences remain across physically valid consistent full states.Source: §B.2.2 · p. 26
• If the removed state is adjacent to another 1-sufficient state, the shared distinguishing atom is assigned false to prevent a true answer from requiring another question.If that atom were true, the state would lead to another 1-sufficient partial state and thus require at least one further question.Source: §B.2.2 · p. 26
• A single possible optimal sequence means no question is needed; with two, attributes that differ between the two state groups provide questions that distinguish their trajectories.Passing the checks means asking about the removed atom fully determines a unique optimal trajectory.Source: §B.2.2 · p. 26

B.3. GSME-Q

• For GSME-Q, annotators first screened out ambiguous or mismatched problems, then produced CSPs that were automatically checked for correct answers; all valid CSPs, including alternate parses, were retained.Three annotators supplied CSPs for each problem, with natural-number domains assumed.Source: §B.3 · p. 27

B.4. GSM-Q

• For GSM-Q, annotators validated equation translations and answers, then removed stated equation premises from the word problem and judged whether the answer changed or became unclear.An equation implied but not explicitly stated in the prompt was marked “cannot remove.”Source: §B.4 · pp. 32–33

C.1. Identifying missing information and asking the right question

• Logic-Q prompts ask the model to choose an attribute question that maximizes information regardless of its answer, or to end questioning once enough information is available.The required question format is “Question: Is Alice [attribute]?”Source: §C.1 · p. 35
• The Planning-Q prompt presents the planning domain in PDDL and lists the current objects in the problem.Source: §C.1 · p. 35
• In C.1, prompts ask models to decide whether information is sufficient and, if not, select a clarification; GSM-Q asks for a variable by name and symbol, while GSME-Q asks for its value by symbol.Planning-Q likewise asks whether the initial state disambiguates a plan and, if not, which missing information would make the plan unambiguous.Source: §C.1 · pp. 35–36

C.2. Reasoning in the presence of sufficient information

• For sufficient-information tests, Logic-Q asks for a binary yes/no answer, while Planning-Q asks for a plan from the supplied PDDL, objects, initial conditions, and goals.The planning response must be a comma-separated list of actions and arguments.Source: §C.2 · p. 36

C.3. Detecting underspecification and recognizing uncertainty

• In uncertainty tests, Logic-Q, Planning-Q, and GSM-Q/GSME-Q prompts allow models to answer “Not sure” when they cannot determine an answer or plan.The prompts otherwise request a yes/no answer, a plan, or a raw numerical answer, respectively.Source: §C.3 · pp. 37–38

D.1. Logic-Q

• Logic-Q brute force infers consequences of current assignments, then tests unassigned variables in both polarities until one makes the target take opposite values; inference runs in O(|X| + |C|).The expected runtime depends on how many variables are tested before reaching one in the sufficient set, with an additional probabilistic bound described in the passage.Source: §D.1 · p. 39
• Logic-Q backward search tracks conjunctions needed to prove the target and can stop early when a disjunction identifies an initial condition whose value suffices to determine the target.Its search cost depends on depth and expansions; the passage gives a depth-bounded runtime when the required disjunction is found within depth d.Source: §D.1 · p. 39

D.2. Planning-Q

• Planning-Q brute force samples candidate atoms without replacement and validates a choice by solving across states consistent with each polarity; validity requires stable shortest paths within each polarity and different paths across polarities.With breadth-first search, the passage bounds validation complexity by O(2^(2|X|)b^2), where b is the number of blocks.Source: §D.2 · pp. 39–40
• Planning-Q backward search replaces per-state breadth-first searches with one search backward from the goal, grouping initial states that share a path and finding the attribute that distinguishes their partial states.It searches over partial states, where each proposition may be true, false, or unknown, and can bound the search by maximum path depth d.Source: §D.2 · p. 40

E. Qualitative Study of QuestBench Reasoning Traces

• The analyses include Gemini 2.5 Pro traces on one Logic-Q and one Planning-Q problem; it was Google’s newest reasoning model as of October 2025.The Gemini 2.0 Flash Thinking Experimental model used during the original evaluations was no longer available when these analyses were conducted.Source: E. Qualitative Study of QuestBench… · p. 41

E.1. Logic-Q

• The prompt requires models to ask an attribute question that provides the most information regardless of its answer, or stop when the target is decidable.The model must output either a question in the specified form or “End questioning.”Source: §E.1 · p. 42
• Gemini asks whether Alice is precious, but the accepted questions are whether she is different, frightened, or hurt.The target is whether Alice is tired, and the example disallows asking whether she is sensible.Source: §E.1 · pp. 42, 44
• Gemini’s trace shifts from forward deduction to backward breadth-first search, then chooses precious as a quick route to proving Alice is tired.It says it has explored the rules and prioritized precious, though the authors note the full trace is unavailable to verify that claim.Source: §E.1 · p. 44
• By pursuing only what establishes tired as true, Gemini misses that precious=false cannot establish tired as false, unlike the accepted questions, which settle the target either way.The authors infer that the model does not investigate the negative direction of the target’s truth conditions.Source: §E.1 · pp. 44–45

E.2. Planning-Q

• Planning-Q asks models to decide whether partially observed initial states provide enough information for an unambiguous plan, choosing a clarification from multiple-choice options.The example supplies known state facts and goal conditions for a blocks-world problem.Source: §E.2 · p. 46
• In the example, uncertainty about whether d is on the table or c is clear is linked: resolving either uncertainty resolves the other.The reasoning trace says the plan must work across the possible initial-state scenarios.Source: §E.2 · pp. 47–48
• The model correctly selects “Is (clear c) true?” as the minimal resolving question, apparently by visualizing the blocks rather than searching.The authors note that this may explain why LLM performance only weakly correlated with search-complexity factors in this domain.Source: §E.2 · p. 48

F. Computational Resources

• Generating Logic-Q and Planning-Q datasets mostly uses CPU resources for backward search, with search and filtering taking three to seven days per domain.Source: F. Computational Resources · p. 49
• Experiments queried Gemini, GPT-4o, and Claude through APIs, while Gemma models ran on TPU v5e node pools or one to two H100 GPUs.Source: F. Computational Resources · p. 49
• Evaluation took a few hours for Logic-Q, up to a few days for Planning-Q, and a few minutes for GSM-Q and GSME-Q.Source: F. Computational Resources · p. 49

G. Limitations

• QuestBench tests only 1-sufficient CSPs, while real-world problems may involve multiple missing facts or more complex dependencies.The benchmark is a first attempt at underspecification that favors formalism and ease of evaluation over generality.Source: G. Limitations · p. 49
• Its formally chosen domains may not capture the naturalistic language understanding and ambiguity found in everyday tasks.Source: G. Limitations · p. 49
• Multiple-choice question selection simplifies evaluation and consistency but may overestimate performance relative to open-ended question generation.Source: G. Limitations · p. 49

H. Broader Impacts

• Improved clarification behavior could support interactive, trustworthy AI in education, healthcare, and scientific research by reducing misunderstandings and improving decisions.Source: H. Broader Impacts · p. 49
• QuestBench provides a defined setting to study model uncertainty and epistemic humility, qualities relevant to alignment and safety.Source: H. Broader Impacts · p. 49
• The evaluated capabilities could enable automated persuasion that elicits personal information, although QuestBench itself does not train open-ended question asking.Source: H. Broader Impacts · p. 49
• Providing rationales for model queries may give users greater control over what information they share.Source: H. Broader Impacts · p. 49