Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

• A capable model should recognize incomplete context, acquire the missing information, and answer only when it determines a unique target.Final task success alone can overstate information-seeking ability because models may answer correctly from incomplete information or fail for query-level reasons.Source: §1 · pp. 1–2
• The evaluation measures queried variables, stopping decisions, and final sufficiency separately from answer generation, addressing ambiguities in existing success- or dialogue-based benchmarks.Constraint structure identifies which variables are needed, while oracle interactions reveal how each queried value changes the feasible assignments.Source: §1 · pp. 1–3
• The paper formalizes multi-turn information seeking as solving k-underspecified CSPs, where k variables are jointly required to determine the target and define when to stop.This formulation supplies ground truth for needed variables and sufficiency, while distinguishing underspecification from ambiguity.Source: §1 · pp. 1–2
• MT-INFOSEEK contains 5,251 problems and 9,006 task instances spanning mathematics, logic, biology, medicine, and general knowledge.Problem families share initial information but vary the hidden target value, preventing success through guessing the most likely target.Source: §1 · p. 2, Figure 1 · p. 2
• Models underestimate missing information, stop before sufficiency, and suffer from incorrect query ordering, while final accuracy also varies after identical interactions.Models that continue searching can reach sufficiency at nearly the rate of models querying correctly first, but ordered dependencies make query order consequential.Source: §1 · p. 2

• Existing interactive LLM studies cover RL-based interaction [8] [9], collaboration [10], decision-oriented games [11] [12], and medical dialogue [4] [13].Source: §2 · p. 3
• Clarification research addresses ambiguity through question generation [14] [15] [16], Bayesian question selection [6] [17], and proactive retrieval-augmented clarification.Source: §2 · p. 3
• Unlike existing benchmarks [4] [6] [19], this work evaluates whether models identify missing variables, acquire them sequentially, and stop when they determine the target.QuestBench provides ground-truth missing variables in an underspecified CSP, while this work extends the formulation to multi-turn acquisition; Appendix Table 2 compares benchmarks.Source: §2 · p. 3

3.1 Problem Formulation

• The framework represents each problem with variables, domains, constraints, partial observations, and a target, defining underspecification by the minimal sufficient query set k.The feasible space contains all assignments consistent with the constraints and observed information; interaction narrows this space until the target is uniquely determined.Source: §3.1 · p. 3
• Problem-level k is a worst-case requirement, while a particular task instance may need fewer queries to distinguish its hidden target.For k > 1, models query unassigned non-target variables sequentially, receive oracle values, and decide when to answer.Source: §3.1 · p. 4

3.2 Multi-Turn Evaluation Protocol

• The evaluation separates k prediction, minimal sufficient-set identification, and sequential task solving, measuring both information acquisition and stopping decisions.Metrics include k over- and underestimation, exact-set accuracy, Jaccard similarity, final answer accuracy, final sufficiency, turn count, and query coverage, correctness, and ordering.Source: §3.2 · p. 4

3.3 Datasets

• MT-INFOSEEK covers mathematics, logic, biology, and medicine as structured domains with validated k-MSS problems, plus open-ended 20Q for free-form question generation.Structured problems use k ∈ {1, 2, 3, 4} where feasible; 20Q has no predefined k or MSS and evaluates dynamically generated questions over candidate sets.Source: §3.3 · p. 5

RQ1

• Models detect underspecification but underestimate its degree: at k=2, they predict a smaller k more than 4× as often as a larger one, while k-prediction accuracy falls below 0.5 for k ≥2.Models predicted k=0 for only 0.6±0.8% of problems, indicating that they rarely treat underspecified tasks as fully specified.Source: §4 · p. 6, Figure 3 · p. 6
• Identifying missing variables is harder than recognizing underspecification: no model exceeds 0.4 exact-set accuracy at k=2 or 0.2 at k=3, and revealing the true k yields only modest improvement.Exact-set accuracy and Jaccard similarity remain low for k≥2, showing that models often fail to select a minimal sufficient set.Source: §4 · pp. 6–7, Figure 4 · p. 7
• Under a fixed query budget, spreading questions across more turns often improves final sufficiency, while sustained later-turn search is especially important as underspecification increases.Early-turn correctness correlates with final sufficiency at k=1 (r=0.60) but not k=3 (r=−0.05), whereas late-turn correctness remains correlated (r=0.46–0.48) and turn count rises from r=0.19→0.66 as k increases.Source: §4 · p. 7
• An initially incorrect query is recoverable when models continue searching: for Logic-Q-MT, recovery after a first-turn miss reaches near-parity with first-turn hits for k=2 and k=3, while first-turn MSS mentions predict recovery and final sufficiency.The gap is within 5 percentage points for 12/14 models at both settings, and MSS relative mention rates have log odds ratios of 0.365 for second-turn recovery and 0.528 for final sufficiency.Source: §4 · p. 8, Figure 6 · p. 8

RQ2

• In dependency-ordered clinical tasks, correct query order consistently improves final accuracy even when all necessary variables are eventually acquired; coverage is the strongest predictor and order correctness is generally second.The order effect strengthens with larger k and more distractors and persists in an open-book setting that supplies the diagnostic algorithms verbatim.Source: §4 · p. 8, Figure 7 · p. 9

RQ3

• Informative questions help but do not guarantee target identification: models with similar expected information gain differ in final accuracy, and stronger models better integrate identical completed dialogues and continue reducing residual uncertainty.Entropy generally decreases across turns but often plateaus later; stronger models maintain lower entropy, and partially informative or high-pass answers can still support later narrowing when models update rather than discard them.Source: §4 · pp. 8–10, Figure 8 · p. 9

5 Discussion and Conclusion

• Multi-turn information seeking should be evaluated by whether models recognize insufficient context, identify needed variables, acquire them, and achieve final sufficiency—not by final accuracy alone.MT-InfoSeek separates answer generation from acquiring information that determines the answer.Source: §5 · p. 10
• Future work should model ambiguous, noisy, changing, or inconsistent responses, probabilistic uncertainty, longer conversations, missing rules or constraints, and unequal query costs [27] [28].Unequal costs would make query selection a budgeted decision, while human-audited protocols and reasoning-trace analyses could further test reported failures.Source: §5 · p. 10

A Ethics statement and broader impacts

• The evaluation suite supports transparent uncertainty handling by measuring missing-information recognition, appropriate follow-up questions, and avoidance of unsupported guesses without collecting private user data.These measurements are intended to support more cautious interactive systems.Source: A Ethics statement and broader impacts · p. 14
• The paper cautions that MT-InfoSeek results are controlled diagnostic measurements, not evidence of clinical or scientific deployment readiness.Clinical decision procedures and biological networks are used only for controlled evaluation, and the suite may reflect cultural, linguistic, or domain-specific assumptions.Source: A Ethics statement and broader impacts · p. 14

B.1 Underspecification and the inference gap

• Underspecification differs from ambiguity because the problem is clear while case-specific information needed to solve it is withheld.Unlike ambiguity, underspecification may leave no visible trace and can prompt models to silently assign presumptive variable values.Source: §B.1 · p. 15
• Bridging the inference gap requires recognizing that the target is undetermined and selecting an action expected to narrow the remaining possibilities, as in clinical testing or scientific experimentation.The paper evaluates this recognition and selection step, while simplifying away acquisition costs and cases where no sufficient evidence is available.Source: §B.1 · p. 15

B.2 LLMs can still achieve high accuracy even with limited information

• The limited-information medical evaluation compares Qwen3-30B-A3B-Thinking-FP8 with full-information prompting without user interaction, requiring a choice in at least 2/4 repeats and forbidding questions or explanations.The broader benchmark comparison tracks multi-turn interaction, proactive questioning, objective query labels, timing or order, and target control.Source: Table 1 · p. 15, §B.2 · pp. 15–16, Table 2 · p. 16

C.1 Formal definitions

• A problem is represented as variables, domains, constraints, a known partial assignment, and a target variable, with the target known only when all feasible assignments agree on its value.Underspecification means the target cannot be determined from the current information alone.Source: §C.1 · p. 16
• The degree k of underspecification is the cardinality of a globally minimal sufficient variable set, while task-level underspecification can be smaller because the hidden target value enables shortcuts.A single problem induces task instances for compatible target values, and a task-specific shortcut need not belong to any problem-level minimal sufficient set.Source: §C.1 · pp. 16–17

C.2 Multi-turn interaction formalization

• Each interaction begins with the problem, alternates model reasoning and variable queries answered by an information source, and terminates when the model outputs a target prediction instead of querying.The interaction history records each reasoning trace, query, and response before the final reasoning trace and prediction.Source: §C.2 · p. 17

C.3 Evaluation metrics

• Evaluation separates degree and missing-variable identification from task solving, measuring exact or Jaccard set agreement, final sufficiency, task accuracy, turns to sufficiency, query coverage and correctness, and query order.Ordered ClinGuide-MT queries are scored with normalized Damerau–Levenshtein similarity against the sufficient sequence.Source: §C.3 · pp. 17–18

C.4 Information source and oracle design

• The evaluation uses adversarial, random, and cooperative oracles; controlled tasks primarily use adversarial oracles that provide constraint-consistent answers while greedily delaying target identification.Adversarial and cooperative oracles use one-step lookahead, random oracles may answer “not sure,” and cooperative evaluations are easier than random or adversarial ones.Source: §C.4 · p. 19

C.6 Reliability of LLM-based oracles

• LLM-based oracle responses were highly consistent and cross-oracle agreement was 93.4%–94.1% for 20Q and 94.0% for ClinGuide-MT, with disagreements concentrated around “Not sure” or other category boundaries.Within-oracle agreement was 91.9%–95.6% for 20Q and 94.0%–97.7% for ClinGuide-MT; most inconsistencies involved uncertainty labels rather than substantive answer reversals.Source: §C.6 · pp. 21–22, Table 4 · p. 21, Table 5 · p. 21
• 20Q oracle agreement with an independent reference panel was 99.9% across 1,669 audited turns, indicating oracle errors cannot explain the observed failure rate or main comparative conclusions.In ClinGuide-MT, manual auditing found 7% clear mismatches and 5% borderline matches, mainly from overmatching related but non-equivalent questions.Source: §C.6 · p. 22, Table 9 · p. 23, Table 10 · p. 23

C.5 Robustness to oracle response policies

• Final sufficiency declined from k = 1 to k = 4 by 29.7–41.9 pp under random and 25.0–39.0 pp under cooperative oracles across all four models.The turn-budget effect and recovery after a first-turn miss also persisted, while cooperative responses produced the highest pooled final sufficiency without changing qualitative conclusions.Source: §C.5 · p. 21

D Dataset construction details

• Table 11 reports dataset statistics by degree of underspecification, including total problem and task counts and counts for each k.Source: Table 11 · p. 24, D Dataset construction details · p. 24

D.1 Logic-Q-MT

• Logic-Q-MT uses recursive candidate generation plus feasibility-aware validation because pairwise flips cannot capture all minimal k-sufficient sets, including XOR/XNOR dependencies.Validation checks sufficiency, variable essentiality over feasible assignments, and global minimality; Horn-SAT implements feasibility and entailment tests.Source: §D.1 · pp. 24–26
• The Logic-Q-MT construction is not complete for arbitrary non-Cartesian feasible domains: a globally minimal 4-sufficient instance may have no conditioning assignment yielding a globally minimal 3-sufficient problem.This blocks complete coverage by recursive k = 3 →4 construction.Source: §D.1 · pp. 25, 27

D.2.1 GSME-Q-MT construction

• GSME-Q-MT masks base variables and retains held-out sets only when the visible context is insufficient, the full set is sufficient, and every smaller subset is insufficient.Candidate query sets include true missing variables, distractors, and decoys, requiring models to distinguish necessary from irrelevant or redundant information.Source: §D.2.1 · pp. 27–28

D.2.2 GSME-Q-MT-Ext construction

• GSME-Q-MT-Ext addresses GSME-Q-MT’s weak interactions and short structures by generating tasks from DAG programs with controllable variables, constraints, reasoning depth, branch merges, and distractors.The construction samples relevant leaf-variable masks, validates sufficiency and minimality, ranks difficulty, and emits masked problem instances with candidate query sets.Source: §D.2.2 · pp. 28–29
• GSME-Q-MT-Ext generates validated arithmetic DAG tasks with backbone, branch, merge, and distractor structures controlling goal-relevant leaves and missing-information level k.Values are assigned by topological execution, and instances are retained only when structural, arithmetic, acyclicity, solvability, and minimal k-sufficiency checks pass.Source: §D.2.2 · pp. 29–30

D.3 GeneReg-MT

• GeneReg-MT uses Boolean gene-regulatory networks to test steady-state and marker-value identification, yielding 800 problems from 38 networks and 2,326 tasks.The construction filters networks to 6–30 nodes and verifies minimal k-sufficient assignments by dropping values or recursively constructing k = 1, 2, 3, 4 tasks.Source: §D.3 · pp. 30–31

D.4.1 ClinGuide-MT problem formulation and construction

• ClinGuide-MT converts 59 guideline-based diagnostic algorithms across 11 clinical areas into ordered-path tasks whose masked clinical variables determine final recommendations.Models receive the observed context and query space without the decision tree, with distractor variables expanding the query set while leaving outcomes unchanged.Source: §D.4.1 · pp. 32–33

D.5.1 Problem formulation

• 20Q formulates open-ended target identification as a generalized underspecified CSP, where each question-answer pair constrains feasible targets but fixed minimal k-sufficient sets do not apply.The model asks yes/no questions or produces a final guess, while the examiner may answer pass for ambiguous or insufficiently grounded questions.Source: §D.5.1 · p. 33

D.5.2 Candidate-space surrogate

• The offline 20Q evaluation approximates the open-ended hypothesis space with a finite candidate set, initializes uniform beliefs, and updates them after each answer.Candidate pools include COMMON and THING subsets, with category-level pools retained for analysis.Source: §D.5.2 · pp. 33–34

D.5.3 Question-induced partitions

• Each question partitions candidates into yes, no, and pass subsets, so informativeness depends on how substantially it reduces current uncertainty.Broad questions can be useful early, whereas specific questions may become more informative after the candidate space narrows.Source: §D.5.3 · p. 34

D.5.4 Belief update

• Beliefs are updated with a soft likelihood model that heavily favors answer-consistent candidates, penalizes inconsistent ones, and downweights pass cases less aggressively.The soft update reflects noisy natural-language answers and treats pass as ambiguity or insufficient grounding rather than contradiction.Source: §D.5.4 · p. 34

D.5.5 Evaluation metrics for 20Q

• 20Q evaluation separates final target accuracy from information seeking, measuring turns, candidate-space entropy, and expected information gain as the mutual information I(Y ; At | qt).Entropy captures remaining uncertainty, while expected information gain measures the anticipated reduction before the answer is observed.Source: §D.5.5 · p. 35
• Normalized expected information gain measures the fraction of current uncertainty a question is expected to remove, while entropy curves and pass mass assess narrowing efficiency and ambiguity.Pass mass is computed before belief updating; high values indicate questions that fail to cleanly partition the plausible candidate space.Source: §D.5.5 · p. 35

D.5.6 Offline question-informativeness evaluation

• Offline replay separates final task success from question informativeness and diagnoses whether models ask high-gain questions, avoid ambiguity, and progressively narrow candidates.The offline analysis replays completed trajectories over a candidate-space surrogate without affecting the model’s interaction protocol.Source: §D.5.6 · p. 36

D.5.7 Human validation of normalized expected information gain

• The study tested whether nEIG tracks human judgments of uncertainty reduction using 200 interaction states drawn from 111 COMMON episodes.Source: §D.5.7 · p. 36
• Each comparison paired a Qwen3-30B-A3B-Thinking-FP8 question with a GPT-5-mini question generated from the same truncated history.Identical questions and pairs with |∆nEIG| < 0.01 were excluded, with at most four separated pairs retained per episode.Source: §D.5.7 · p. 36
• A blinded graduate-level annotator selected which question would better narrow remaining possibilities after viewing the candidate pool, prior history, and randomized candidate questions.The annotator did not see the hidden target, candidate answers, model identities, or nEIG scores.Source: §D.5.7 · p. 36
• 153 of 200 pairs (76.5%) favored the higher-nEIG question, significantly exceeding the 50% chance rate across nEIG-difference bins.Table 13 stratifies agreement by absolute nEIG difference and reports significance against chance in every bin.Source: §D.5.7 · p. 36, Table 13 · p. 36
• Agreement rose to 86.5% at confidence level 3, versus 70.6% and 70.7% at confidence levels 2 and 1.The annotator’s confidence was recorded on a three-point scale.Source: §D.5.7 · p. 36
• The evidence supports nEIG as aligned with human judgments of reducing uncertainty over remaining candidates, not as a measure of holistic question informativeness.nEIG depends on the surrogate candidate pool, belief-update procedure, and offline evaluator described in Appendix D.5.5, so its values inherit their assumptions.Source: §D.5.7 · p. 36

Serving infrastructure

• Open-weight models run locally through containerized vLLM servers, while closed-API models use their provider APIs through the same evaluator.Evaluation begins only after liveness and model-readiness checks succeed for local servers.Source: §E.2 · pp. 44–45

GPU allocation

• Open-weight inference uses H100 GPUs with tensor parallelism set to TP=4 for Qwen3.5-122B-A10B-FP8, TP=2 for Qwen3-Next-80B-A3B-Thinking-FP8, and TP=1 for all other studied models.The remaining models include Qwen3.5-{4B, 9B, 27B-FP8, 35B-A3B-FP8}, Qwen3-4B-Thinking-2507, Qwen3-30B-A3B-Thinking-2507-FP8, and gpt-oss-{20B, 120B}.Source: §E.2 · p. 45

Inference configuration

• All servers use a 131,072-token maximum context, supported FP8 KV caching, chunked prefill, asynchronous scheduling, and disabled multimodal inputs and prefix caching for consistency.Memory utilization is 0.95 for single-GPU servers and 0.90 for tensor-parallel servers, with batching and evaluator concurrency tuned to saturate each server without exceeding its sequence cap.Source: §E.2 · p. 45
• For gpt-oss models, reasoning effort is set to medium or high according to the corresponding results, while reasoning-parser settings follow model-family defaults.Source: §E.2 · p. 45

F Benchmarking results

• Open-weight models run locally on Nvidia H100 GPUs with vLLM, whereas proprietary models use provider APIs with their recommended default inference parameters.This benchmarking setup applies across the reported evaluations.Source: F Benchmarking results · p. 46
• Models vary substantially in final sufficiency: GPT-5 and GPT-5-mini query all required information in almost all Logic-Q-MT cases and many GeneReg-MT cases, while others usually fall short.These results cover Logic-Q-MT, GeneReg-MT, GSME-Q-MT, and GSME-Q-MT-ext, with full results reported in Tables 14–16.Source: F Benchmarking results · p. 46
• Models generally perform worse on tasks with higher k, despite receiving no prompt information about the degree of underspecification.The comparison is reported across Logic-Q-MT, GeneReg-MT, GSME-Q-MT, and GSME-Q-MT-ext.Source: F Benchmarking results · p. 46
• Logic-Q-MT results compare final sufficiency, answer accuracy, average turns, and over-questioning across underspecification levels k.Table 14 excludes Gemini-3.1-Pro from cross-model analyses because it was evaluated on a reduced set of conditions.Source: Table 14 · p. 47
• GeneReg-MT results report final sufficiency, answer accuracy, average turns, and over-questioning for the benchmark’s task conditions.Table 15 likewise excludes Gemini-3.1-Pro from cross-model analyses because it was evaluated on a reduced set of conditions.Source: Table 15 · p. 48
• GSME-Q-MT and its extension are evaluated using final sufficiency, answer accuracy, average turns, and over-questioning.The corresponding results are summarized in Table 16.Source: Table 16 · p. 49
• ClinGuide-MT results vary the number of distractor-variable sets, denoted #d, with distractors drawn from filtered variables in other ClinGuide-MT problems.Results are reported in Table 17, where d denotes the distractor-variable set.Source: F Benchmarking results · p. 46, Table 17 · p. 49
• 20-Questions results cover the Common and Things domains, with outcomes reported in Table 18.The supplied passage identifies the table and domains but does not provide cell values.Source: F Benchmarking results · p. 46, Table 18 · p. 50

Feature extraction

• The analysis defines MSS relative mention rate as the fraction of MSS variables mentioned in strict query-planning contexts, contrasting it with generic awareness of underspecification.Outcomes are final sufficiency and second-turn hit, restricted to interactions whose first query misses the MSS and continue to a second turn.Source: §G.1.1 · p. 51

Stability analysis

• Stability refits within model-by-underspecification cells support the Logic-Q-MT association, whereas GeneReg-MT recovery is limited by cells with fewer than 50 usable interactions or other estimation failures.Controls include first-turn CoT length, chosen-query mention, search breadth, and task-specific problem-difficulty variables.Source: §G.1.1 · pp. 51–52

Logic-Q-MT results

• βz = 0.528 (p = 6.71 × 10−45) links MSS relative mention rate to Logic-Q-MT final sufficiency, while awareness is near zero (βz = 0.005, p = 0.853).For second-turn hit after a first-turn miss, the corresponding coefficients are βz = 0.365 (p = 7.99 × 10−23) and βz = 0.010 (p = 0.774).Source: §G.1.1 · p. 52

GeneReg-MT results

• βz = 0.340 and 1.016 predict GeneReg-MT final sufficiency in steady-state and marker tasks, respectively, with positive associations also observed for second-turn hit.Second-turn-hit coefficients are βz = 0.274 for steady-state and βz = 0.590 for marker tasks; stability cells are much sparser than in Logic-Q-MT.Source: §G.1.1 · p. 52

Adjustment sensitivity

• 0.528 to 0.562 is the Logic-Q-MT final-sufficiency change when problem factors are dropped, while other MSS-relative-mention effects also change only modestly.Across analysis frames, VIF ranges from 1.158 to 1.563, indicating the cue is not collinear with the MSS-incidence control.Source: §G.1.1 · p. 52

Annotated examples

• Qwen3-4B-Thinking traces thoughtless through antecedent attributes, asking about weary and then straightforward, matching MSS {straightforward, weary}.Its first-turn query was exuberant, and the CoT explicitly links weary to a thoughtless rule before checking whether Alice is distinct and straightforward.Source: §G.1.2 · p. 52

CoT predictors

• MSS relative mention rate is the strongest CoT predictor of final sufficiency and second-turn hit in Logic-Q-MT, while underspecification awareness is near zero.The standardized log-odds ratios are βz = 0.528 and 0.365 for MSS relative mention rate, versus βz = 0.005 and 0.010 for underspecification awareness.Source: Figure 18 · p. 53
• MSS relative mention rate remains positive for final sufficiency and second-turn recovery in both GeneReg-MT steady-state and marker tasks.For final sufficiency, βz = 0.340 for steady state and βz = 1.016 for marker; for second-turn hit, βz = 0.274 and 0.590.Source: Figure 20 · p. 54

Annotated examples

• Qwen3-30B-A3B-Thinking-FP8 identifies the rule plain and proud -> smart, but must choose between querying proud and plain before checking the other attribute.The target is smart; its queries are proud then plain, while the canonical MSS is {long, plain, thoughtless}.Source: §G.1.2 · p. 55
• Cited examples show Qwen3.5-122B-A10B-FP8 querying talented then mean for gifted (MSS = {careless, mean}), while gpt-oss-20B-high queries versatile then busy for elegant (MSS = {busy, sincere, uptight}).Source: §G.1.2 · p. 55

G.1.3 Additional results

• No-budget Logic-Q-MT interactions can reach the 10-turn cap, motivating tests of whether explicit rejection of answers issued before k queries corrects premature stopping.Figure 22 reports the proportion reaching the cap, while the intervention tests a weaker correction than forcing sufficiency.Source: Figure 22 · p. 55, §G.1.3 · p. 55

G.1.4 Prompt sensitivity

• Prompt-sensitivity experiments compare rule and fact reordering, query-selection guidance, and insufficiency reminders across Logic-Q-MT and GeneReg-MT using four representative models.The study calibrates consequential changes against rerun noise with problem-level cluster bootstrap tests using 10,000 resamples and Holm correction.Source: §G.1.4 · p. 59
• Prompt sensitivity is confined mainly to content-bearing edits and weaker models, while GPT-5-mini remains robust across variants except for −0.040 under P2 in Logic-Q-MT.P1 reordering stays within run-to-run noise, P2 effects are small and model-dependent, and P3 removal of the insufficiency reminder lowers final sufficiency, especially for smaller open-weight models.Source: §G.1.4 · pp. 59, 65, Table 22 · p. 65
• The evaluation uses fixed, shared prompts and adopts no budget by default because stating a turn budget can alter the stopping behavior under study.The k-prediction and MSS prompts were minimally adapted from QuestBench, with improved prompting strategies left to future work.Source: §G.1.4 · p. 65

G.1.5 Preventing premature answers

• Rejecting answers before k queries prevents clear under-querying but does not constrain query selection or guarantee sufficiency, with the intervention evaluated at k = 3.The study focuses on k = 3 because higher underspecification is associated with more premature stopping.Source: §G.1.5 · p. 65
• +25.7 pp (p < 5 × 10−4) and +9.7 pp (p = 0.003) final-sufficiency gains occur for Qwen3-30B-A3B-Thinking-FP8 and Qwen3-4B-Thinking when premature answers are rejected at k = 3.Effects for gpt-oss-20B-high and GPT-5-mini are insignificant; the intervention also increases turn counts beyond the k-query minimum.Source: §G.1.5 · p. 66, Table 23 · p. 66
• The intervention’s gains indicate a latent capability elicited by explicit correction rather than spontaneous calibration, because the protocol leaks k and enforces continued querying.Rejecting early answers may signal that current information is insufficient, making models more cautious afterward.Source: §G.1.5 · p. 66

G.1.6 Sensitivity to the designated MSS

• Logic-Q-MT has substantial MSS multiplicity: 96.8% of problems have multiple MSSs, averaging 5.98, 8.74, and 10.34 MSSs for k = 1, 2, and 3.Final sufficiency is invariant to MSS choice, but the queryable variables depend on the designated MSS and can affect model behavior.Source: §G.1.6 · p. 66
• Final sufficiency rises by up to 7.2 percentage points at k = 3 for alternative-MSS variants, with gains in 7 of 8 comparisons and slightly fewer turns.Most differences fall within run-to-run noise across the 16 noise-anchored comparisons.Source: §G.1.6 · p. 66, Table 25 · p. 67, Table 26 · p. 67
• Only Qwen3-4B-Thinking’s turn reduction on altgt2 survives Holm correction; two additional comparisons are significant before correction.The uncorrected comparisons are Qwen3-4B-Thinking’s turn reduction on altgt1 and gpt-oss-20B-high’s final-sufficiency shift on altgt1.Source: §G.1.6 · p. 67
• Alternative MSSs are easier because refutation can establish sufficiency, whereas original canonical MSSs were certified by forward chaining and are systematically deeper.At k = 3, all original MSSs are forward-derivable with mean depth 4.40, versus roughly 60% of resampled draws with mean depth 3.5.Source: §G.1.6 · p. 67, Table 27 · p. 67

G.2 GSME-Q-MT

• Several models perform well when missing information is requested in one turn but become less reliable when the same information must be acquired interactively.The gap is small on GSME-Q-MT, where instances are relatively simple and many models remain near saturation.Source: §G.2 · p. 68, Figure 39 · p. 68
• GSME-Q-MT-Ext reveals increasingly large multi-turn accuracy drops for Qwen3-4B-Thinking and Qwen3-30B-A3B-Thinking-FP8 as k grows.The pattern suggests that the challenge is managing the information-seeking process required to fully specify the arithmetic problem, rather than solving the arithmetic itself.Source: §G.2 · p. 68, Figure 38 · p. 68
• GPT-5-mini remains stable across single-turn and multi-turn settings, while Gemini-3-Flash changes only slightly.These results indicate that some models better preserve their single-turn planning ability during interaction.Source: §G.2 · p. 68, Figure 39 · p. 68

G.3 ClinGuide-MT full results

• Achieving high ClinGuide-MT final accuracy requires asking the correct questions in the correct order.The full results support the conclusion that information acquisition quality affects downstream reasoning beyond whether information is eventually acquired.Source: §G.3 · pp. 68–69
• Regression analysis finds query coverage the strongest and most stable positive predictor of final accuracy, with order correctness usually second-ranked and query correctness additionally beneficial.Query correctness becomes more salient in harder settings by capturing avoidance of off-branch or irrelevant queries.Source: §G.3 · pp. 68–69
• Full MSS coverage substantially improves final accuracy, while clinically induced query ordering generally provides additional gains, especially for larger k and more distractors.Incorrect ordering can reduce accuracy even when all MSS variables are eventually covered.Source: §G.3 · p. 68, Figure 40 · p. 69

G.3.1 Open-book evaluation

• The open-book evaluation provides all 59 source-textbook diagnostic algorithms [21] in their original wording, including the algorithm relevant to each case.The decision trees specify procedures, while patient-specific pathway values must still be acquired through questioning.Source: §G.3.1 · p. 69, Table 28 · p. 70
• Open-book access to all 59 diagnostic algorithms substantially improves final accuracy across four models and both difficulty levels, with the largest gains for Qwen3-4B-Thinking.Accuracy nevertheless remains far from saturated, showing that access to the relevant algorithm alone does not solve the task.Source: §G.3.1 · p. 69, Table 28 · p. 70
• Full MSS coverage yields the highest final accuracy in the open-book setting wherever the strata are populated.The comparison is stratified by MSS coverage in Table 29.Source: §G.3.1 · p. 69, Table 29 · p. 70
• Following the diagnostic algorithm’s suggested query order further boosts final accuracy in the open-book setting wherever the strata are populated.Coverage and ordering effects carry over from the closed-book setting.Source: §G.3.1 · p. 69, Table 29 · p. 70
• This open-book setup is favorable and near-upper-bound because the exact task wording and clear decision-tree structure reduce solving largely to matching each case to the correct tree and node.Its gains likely overstate real-world open-book performance because real knowledge sources may be larger or noisier.Source: §G.3.1 · p. 71
• Residual errors in the favorable open-book setting reflect failures to identify, order, and acquire case-specific information rather than missing medical knowledge.Accuracy remains unsaturated even though all diagnostic trees are provided, and coverage and ordering effects persist.Source: §G.3.1 · pp. 69, 71

G.4.1 Full results

• Higher question informativeness generally corresponds to higher final accuracy, but question quality alone does not determine 20Q success.The relationship is especially clear under the GPT-5-mini examiner, whereas some Qwen3-30B-A3B-Instruct-FP8 results retain competitive accuracy despite similar or lower question-quality scores.Source: §G.4.1 · p. 71, Figure 42 · p. 71
• Final accuracy also depends on integrating accumulated interaction history and using remaining evidence to make a correct final guess.Thus, offline question informativeness captures an important but incomplete aspect of multi-turn information seeking.Source: §G.4.1 · p. 71
• Entropy decreases across turns and settings, confirming that interaction progressively narrows the candidate space.This pattern holds across both datasets, both examiners, and almost all guesser models.Source: §G.4.1 · pp. 71–72, Figure 43 · p. 71
• Later turns usually yield smaller entropy reductions because models shift from broad categorical questions to weaker fine-grained follow-ups.The results indicate that sustaining discriminative questioning after broad categories are identified is a central 20Q challenge.Source: Figure 43 · p. 71, §G.4.1 · p. 72

G.4.2 Case study

• The case study analyzes four Qwen3-4B-Thinking failures using QA traces, remaining entropy, and pass-support mass.This model was selected because it achieved relatively high question-quality scores but performed noticeably worse at recovering the final target; pass-support mass measures candidate support for ambiguous or non-committal answers.Source: §G.4.2 · p. 72, Figure 44 · p. 72
• The four cases reveal weak final disambiguation, ambiguity accumulation, inconsistent belief-state updates, and search at the wrong abstraction level.Figure 45 presents representative failure modes in 20 questions.Source: §G.4.2 · p. 72, Figure 45 · p. 73
• Across failure patterns, models often narrow the candidate space but fail to consolidate that progress into a correct final answer.Effective information seeking therefore requires coherent search-state maintenance, appropriate use of ambiguous evidence, and refinement at the right semantic granularity.Source: §G.4.2 · p. 72