PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

Yepeng Huang, Jiawen Zhang, Michelle Dai, Xiaorui Su, Shanghua Gao, Zi Wang, Marinka Zitnik

arXiv:2608.14808v1cs.AIcs.CLcs.LG

TL;DR

Existing evaluations do not directly test whether LLMs identify and acquire the information needed to answer underspecified questions. This paper formalizes multi-turn information seeking and evaluates it across domains, finding that models underestimate missing information, query incompletely, and often stop too soon.

  • Problem

    Existing multi-turn benchmarks do not specify which information models should request or whether they stop only once the target becomes identifiable.

  • Method

    The authors formalize information seeking as k-underspecified constraint solving and evaluate querying, stopping, and answer effects in MT-INFOSEEK across five domains.

  • Results

    Across domains, models underestimate underspecification, query incomplete variable sets, and often stop before acquiring sufficient information; for k = 2, underestimation occurs over 4× as often as overestimation.

  • Takeaways & Limitations

    Final accuracy alone does not measure interactive information-seeking competence, so evaluation should separately assess missing-information recognition, query selection, and final sufficiency.

  • Takeaways & Limitations

    The evaluation treats every query as equally expensive and uses an LLM as the oracle in ClinGuide-MT and 20Q.

Abstract

from arXiv · show

When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it, and respond only once that information determines a unique answer. We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem, where k is the number of variables jointly required to determine the target and therefore measures the degree of missing information. We instantiate the formulation in MT-InfoSeek, a controlled evaluation suite of 5,251 problems and 9,006 task instances spanning mathematics, logic, biology, medicine, and general knowledge. We evaluate models along three axes: what they ask, when they ask it, and how the acquired information affects the final answer. Performance degrades across models and domains as underspecification increases. Models recognize that additional information is needed but underestimate how much, and in logical problems at k = 2 they under-predict the degree of missing information about four times as often as they over-predict it. They also fail to identify a minimal sufficient set of queries, improve only marginally when given the true k, and often stop before acquiring sufficient information. In tasks with ordered dependencies, an incorrect query order reduces final accuracy even when the model eventually acquires all necessary information. We measure information seeking directly through final sufficiency, which records whether the acquired information determines the target independent of answer generation. This separation shows differences between models that final accuracy alone does not capture, and indicates that the ability to seek information over multiple turns is distinct from the ability to generate answers and is not measured by current LLM evaluations.

1 Introduction

The paper formalizes multi-turn information seeking as acquiring the missing variables needed to uniquely determine an underspecified task, then evaluates what models ask, when they stop, and how acquired information affects answers. MT-INFOSEEK separates information-seeking sufficiency from final answer accuracy across five domains.

  • Motivation: Underspecified tasks require models to recognize insufficient context, acquire missing information, and answer only once the target is uniquely determined.The paper distinguishes underspecification from ambiguity and focuses on whether acquired information makes the target identifiable.
  • Motivation: Existing multi-turn benchmarks conflate information acquisition with answer generation and do not establish which variables should be queried or when stopping is sufficient.Free-form interaction can obscure whether failure arose from a wrong query, an uninformative response, or a response that does not map to task variables.
  • Evaluation and findings: MT-INFOSEEK evaluates which variables models query, when they stop, and how acquired information affects final answers across mathematics, logic, biology, medicine, and general knowledge.The evaluation measures final sufficiency independently from final accuracy and finds that models underestimate missing information, degrade as k grows, and often stop too early.
  • Formalization: The paper models information seeking as solving a k-underspecified constraint satisfaction problem, where k variables are jointly required to determine the target.This formulation supplies ground truth for needed variables, supports controlled variation in k, and defines when the model should stop asking.

2 Related Work

Prior work studies interactive information seeking and clarification under ambiguity across diverse environments and domains. Existing benchmarks mainly assess final success or dialogue quality, while QuestBench provides the closest underspecified-CSP foundation for the paper’s multi-turn setting.

  • Multi-turn information seeking: Recent studies examine LLM information seeking through interaction, feedback, adaptation, clarification questions, and Bayesian experimental design.These settings include RL-based interaction, collaborative environments, decision-oriented games, and medical dialogue.
  • Benchmarking under incomplete information: Existing benchmarks span clinical, mathematical, logical, and common-knowledge domains but primarily measure final task success, dialogue quality, or judge-mediated outcomes.This evaluation emphasis leaves multi-turn information acquisition less directly measured.
  • Benchmarking under incomplete information: QuestBench is closest to this setting because it supplies ground-truth missing variables through an underspecified CSP formulation.The paper extends that formulation to multi-turn settings in which models acquire information incrementally.

3 MT-INFOSEEK Evaluation Framework

MT-INFOSEEK formalizes multi-turn information seeking as solving underspecified constraint satisfaction problems and evaluates missing-information assessment, query planning, and sequential task solving separately. It instantiates this framework across mathematics, logic, biology, medicine, and general knowledge with metrics that distinguish information sufficiency from final answer generation.

  • CSP formulation: A problem P consists of variables, domains, constraints, an observed partial assignment, and a target variable whose feasible assignments define the remaining possibilities.The feasible space contains all full assignments consistent with the constraints and observed information.
  • CSP formulation: The framework defines underspecification by whether multiple target values remain feasible, and sufficient query sets as variables that determine the target for every valid assignment.The interaction queries unassigned non-target variables until all remaining feasible assignments agree on the target.
  • Evaluation protocol: The evaluation separates k-prediction, minimal sufficient set identification, and sequential task solving to isolate information assessment, query planning, and interactive behavior.The task-level evaluation measures both the final answer and whether acquired information alone determines the target.
  • Evaluation protocol: The benchmark evaluates complete task families so models cannot succeed by guessing likely targets and must narrow the feasible space through queries.Each family shares the observed assignment but varies in the hidden target value.
  • Benchmark construction: MT-INFOSEEK instantiates multi-turn problems across mathematics, logic, biology, medicine, and general knowledge, generating k-underspecified structured tasks for k ∈ {1, 2, 3, 4} where feasible.Structured domains use predefined finite queryable variables, with masked sets verified as valid k-MSSs.

4 Results

Results show that models recognize underspecification but often misjudge and fail to acquire the variables needed for identification, while sustained multi-turn querying can recover early misses. Final success also depends on query order and evidence integration: models narrow candidate spaces but struggle to distinguish the remaining targets.

  • RQ1: Which variables to query?: Models detect missing information but underestimate how many additional variables are needed to identify the target.Final sufficiency measures whether acquired information determines the target independently of the model’s answer.
  • RQ1: Which variables to query?: 0.6±0.8% of problems were classified as fully specified, while k-prediction accuracy fell below 0.5 for k ≥2.Errors were consistently biased toward underestimating missing information.
  • RQ1: Which variables to query?: No model exceeded 0.4 exact-set accuracy at k=2 or 0.2 at k=3, and providing the true k improved performance only modestly.Both exact-set accuracy and Jaccard similarity remained low for k≥2.
  • RQ1: Which variables to query?: Under the same total query budget, spreading queries across more turns often produced higher final sufficiency than allowing more questions per turn.Final sufficiency generally increased as more turns were allowed in Logic-Q-MT and GeneReg-MT.
  • RQ2: When to query?: In order-dependent clinical tasks, violating the correct query order reduced final accuracy even when all necessary variables were eventually acquired.Following the correct order improved final accuracy in both fully and partially MSS-covered interactions, especially at larger k and with more distractors.
  • RQ3: How does acquired information affect the final answer?: Models’ queries can have similar expected information gain yet yield substantially different final accuracy, showing that evidence integration also determines success.Early queries usually reduce candidate uncertainty, but later queries often flatten in their ability to distinguish closely related remaining targets.

5 Discussion and Conclusion … B.2 LLMs can still achieve high accuracy even with limited information

The paper frames multi-turn information seeking as a distinct capability: models must detect underspecification, acquire sufficient information, and avoid unsupported guesses. Its controlled evaluation also highlights limitations in task realism and shows that high accuracy with limited information does not by itself establish interactive competence.

  • 5 Discussion and Conclusion: Models detect missing information but often underestimate its degree, query incomplete variable sets, and stop before the target is determined.Final-answer accuracy alone therefore misses important differences in interactive competence.
  • A Ethics statement and broader impacts: The framework evaluates whether models recognize missing information, ask appropriate follow-up questions, and avoid unsupported guesses rather than measuring only final answers.The authors present this evaluation as supporting more transparent uncertainty communication and greater caution under underspecification.
  • A Ethics statement and broader impacts: MT-INFOSEEK provides diagnostic measurements on controlled tasks, not evidence that models are ready for clinical or scientific deployment.Clinical decision procedures and biological networks are included only for controlled evaluation, while the suite may reflect cultural, linguistic, or domain-specific assumptions.
  • B.1 Underspecification and the inference gap: Underspecification differs from ambiguity because the problem is clear but case-specific information needed to solve it is withheld.Ambiguity concerns uncertainty about which problem the user intends to pose, whereas underspecification concerns missing information about the case.
  • B.1 Underspecification and the inference gap: Underspecification is difficult to detect because it leaves no explicit trace, allowing models to answer confidently by silently assigning unqueried variables presumptive values.Early assumptions in multi-turn interaction can condition subsequent turns and surface only later.
  • B.1 Underspecification and the inference gap: The clinical inference gap is the distance between information known at the point of care and evidence required to decide for an individual patient.It is bridged by acquiring missing information through questions, tests, or clinical studies.
  • B.2 LLMs can still achieve high accuracy even with limited information: Table 1 reports Qwen3-30B-A3B-Thinking-FP8 performance on the full MediQ dataset with limited or full prompt information and no user interaction.The accompanying setup includes a limited-information condition in which the model must choose an answer without asking questions or providing explanations.
  • B.2 LLMs can still achieve high accuracy even with limited information: The paper compares MT-INFOSEEK with existing interactive information-seeking benchmarks across multi-turn interaction, proactive questioning, objective query labels, and timing or order evaluation.The comparison distinguishes full, partial or judge-mediated, and absent direct support across these capabilities.

C Formalization and evaluation details

The section formalizes multi-turn information seeking as a constraint-satisfaction problem with minimal sufficient query sets and defines metrics for underspecification, query selection, task solving, and query order. It also evaluates oracle reliability, finding high within- and cross-oracle agreement and minimal impact from residual errors on reported outcomes.

  • C.1 Formal definitions: Information-seeking problems are represented as CSPs with variables, domains, constraints, known assignments, and a target variable whose value must be determined.The formalization assumes satisfiable constraints and known assignments, with feasible full assignments defining possible target values.
  • C.1 Formal definitions: A target is known when all feasible assignments consistent with the current information give it the same value; otherwise, the problem is underspecified.Underspecification is defined as ¬Known(Y | A).
  • C.1 Formal definitions: A minimal k-sufficient set contains exactly k queryable variables, determines the target, and has no smaller sufficient subset.The unique k measures a problem’s degree of underspecification, while task instances are induced by compatible target values.
  • C.1 Formal definitions: Task-level underspecification can be smaller than problem-level underspecification when the observed target value makes fewer variables sufficient.In the example, a 2-underspecified problem requires only X1 to derive y1 for the task instance Ty1.
  • C.2 Multi-turn interaction formalization: The multi-turn protocol initializes the model with P, has it generate a reasoning trace and variable query each turn, receives an oracle response, updates the history, and then produces a final answer.Queries request values of variables in U(P), and the interaction history records each reasoning trace, query, and response.
  • C.3 Evaluation metrics: Evaluation measures predicted underspecification degree, missing-variable accuracy and Jaccard similarity, task accuracy, final sufficiency, turns to sufficiency, total turns, and query coverage and correctness.Missing-variable metrics compare the predicted set ˆS with a ground-truth minimal sufficient set S, while task-solving metrics assess whether the target is known and correctly answered.
  • C.3 Evaluation metrics: Ordered-query evaluation uses normalized Damerau–Levenshtein similarity between the model’s sufficient-query subsequence and the ground-truth ordered sequence.Order correctness is defined as OrdCorr(ˆQ) = 1 − dDL(ˆQ∗, Q) / max(|ˆQ∗|, |Q|).

C.5 Robustness to oracle response policies … D.5.6 Offline question-informativeness evaluation

The paper validates robustness to oracle response policies and constructs controlled multi-turn datasets across logic, arithmetic, gene regulation, clinical guidelines, and offline question-informativeness evaluation. These resources test whether models acquire minimally sufficient information and distinguish informative questioning from final-answer success.

  • C.5 Robustness to oracle response policies: From k = 1 to k = 4, final sufficiency decreased for all four models under random and cooperative oracles.The decline was 29.7–41.9 pp under the random oracle and 25.0–39.0 pp under the cooperative oracle; 54 of 64 fitted budget slopes were positive.
  • C.5 Robustness to oracle response policies: Cooperative responses produced the highest pooled final sufficiency for all four models, but oracle type did not alter the qualitative conclusions.The robustness analysis reran GeneReg-MT with random and cooperative oracles for Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking-FP8, GPT-5-mini, and gpt-oss-20B-high.
  • D.1 Logic-Q-MT: Logic-Q-MT uses a recursive candidate-construction algorithm followed by feasibility-aware validation because pairwise variable flips cannot identify all minimal k-sufficient sets.XOR/XNOR-style dependencies at k = 2 provide a counterexample to the pairwise criterion.
  • D.2 GSME-Q-MT and GSME-Q-MT-Ext: GSME-Q-MT masks relevant base variables and retains minimal k-sufficient held-out sets, while GSME-Q-MT-Ext adds DAG-based depth, branching, merge nodes, and distractor structure.The extended construction addresses the original dataset’s few variables, short dependency chains, and weak interactions among missing quantities.
  • D.3 GeneReg-MT: GeneReg-MT constructs 800 problems from 38 GRNs, inducing 2,326 tasks with targets ranging from full steady states to marker-gene values.Tasks are built by dropping or recursively constructing minimal sufficient gene-expression assignments.
  • D.4.1 ClinGuide-MT problem formulation and construction: ClinGuide-MT presents observed context and query options without the underlying decision structure, testing whether models identify missing clinical variables needed for recommendations.Clinical guidelines provide ordered conditional decisions for diagnosis, treatment, triage, and follow-up.
  • D.5 20 Questions: The 20 Questions framework represents information seeking through candidate-space partitions and belief updates, with evaluation metrics targeting question quality as well as task success.The supplied passages identify these sections but do not provide their detailed formulations or metric definitions.
  • D.5.6 Offline question-informativeness evaluation: Offline evaluation separates final task success from question informativeness and diagnoses whether models ask high-gain, unambiguous questions that progressively narrow the candidate space.A model can guess correctly after weak questions or fail despite asking informative ones.

D.5.7 Human validation of normalized expected information gain · E Prompts and implementation details

A blinded study found that normalized expected information gain generally tracks human preferences for questions that reduce uncertainty over remaining candidates. The validation supports this narrower property, not holistic informativeness, because nEIG depends on the surrogate candidate pool and belief-update procedure.

  • D.5.7 Human validation of normalized expected information gain: At each state, annotators compared questions from Qwen3-30B-A3B-Thinking-FP8 and GPT-5-mini generated from identical truncated histories.Identical questions, pairs with |∆nEIG| < 0.01, and excess pairs per episode were excluded.
  • D.5.7 Human validation of normalized expected information gain: 153 of 200 pairs (76.5%) favored the higher-nEIG question, exceeding the 50% chance rate.Agreement was significantly above chance in every absolute nEIG-difference bin.
  • D.5.7 Human validation of normalized expected information gain: 86.5% agreement occurred for highest-confidence judgments, versus 70.6% and 70.7% at confidence levels 2 and 1.The confidence pattern indicates stronger alignment when human preferences were expressed with greater confidence.
  • D.5.7 Human validation of normalized expected information gain: The validation directly supports nEIG as a measure of uncertainty reduction over the remaining candidates.This is the property nEIG measures under the current belief and surrogate candidate space.
  • D.5.7 Human validation of normalized expected information gain: The findings do not establish that nEIG reproduces holistic judgments of question informativeness.nEIG is defined relative to the surrogate candidate pool, belief-update procedure, and offline evaluation setup.

E.2 Experiment compute resources

The evaluation uses a uniform multi-turn serving setup: open-weight models run locally on H100 GPUs, while closed-API models use provider APIs through the same evaluator. Servers are containerized, readiness-gated, and configured with standardized inference limits and scheduling choices.

  • Serving infrastructure: Open-weight models run locally with vLLM behind OpenAI-compatible endpoints, while closed-API models are queried through provider APIs using the same evaluator.Each evaluation job launches a dedicated containerized vLLM server on a dynamically selected free port; evaluation starts only after liveness and readiness checks succeed.
  • GPU allocation: Open-weight inference uses H100 GPUs with tensor parallelism set to TP=4, TP=2, or TP=1 according to model size.TP=4 is used for Qwen3.5-122B-A10B-FP8, TP=2 for Qwen3-Next-80B-A3B-Thinking-FP8, and TP=1 for all other listed open-weight models.
  • Inference configuration: All servers use a maximum context length of 131,072 tokens, FP8 KV caching where supported, chunked prefill, and asynchronous scheduling.Multimodal inputs and prefix caching are disabled for consistency.
  • Inference configuration: GPU memory utilization is set to 0.95 for single-GPU servers and 0.90 for tensor-parallel servers.Per-model batching parameters and evaluator concurrency are tuned to saturate each server without exceeding its sequence cap.

F Benchmarking results · G Additional results and experiments · G.1 Logic-Q-MT and GeneReg-MT

Benchmarking across multi-turn information-seeking tasks shows substantial variation in final sufficiency: some models query nearly all required information, while others usually stop too early. Results span Logic-Q-MT, GeneReg-MT, GSME-Q-MT, ClinGuide-MT, and 20-Questions, with final sufficiency separated from answer accuracy and turn usage.

  • F Benchmarking results: Open-weight models run locally on Nvidia H100 GPUs with vLLM, whereas proprietary models run through provider APIs using default recommended inference parameters.This describes the benchmarking setup across the evaluated models.
  • F Benchmarking results: Some models, including GPT-5 and GPT-5-mini, query all required information in almost all Logic-Q-MT cases and many GeneReg-MT cases.Other models usually fail to query sufficiently, as measured by final sufficiency.
  • F Benchmarking results: Final sufficiency is treated as the key metric for evaluating which variables models query in Logic-Q-MT, GeneReg-MT, GSME-Q-MT, and GSME-Q-MT-ext.The benchmark reports varying model performance across these datasets.
  • F Benchmarking results: Logic-Q-MT results report Suff., Acc., Turn, and OverQ across different underspecification levels k.These abbreviations denote final sufficiency, answer accuracy, average turns, and over-questioning rate.
  • F Benchmarking results: GeneReg-MT results report Suff., Acc., # Turn, and OverQ, with Gemini-3.1-Pro evaluated on a reduced set of conditions.Gemini-3.1-Pro is excluded from the cross-model analyses in the main text.
  • F Benchmarking results: GSME-Q-MT and its extension results report Suff., Acc., # Turn, and OverQ.These metrics denote final sufficiency, answer accuracy, average turns, and over-questioning rate.
  • F Benchmarking results: ClinGuide-MT varies distractor variables using filtered variables from other ClinGuide-MT problems, with #d denoting the number of distracting variable sets.Results are reported in Table 17.
  • F Benchmarking results: 20-Questions results cover the Common and Things domains.Results are reported in Table 18.

G.1.1 Details on first-turn CoT analysis

First-turn CoT analysis measures whether models mention variables in the minimal sufficient set (MSS), finding that MSS-relative mention predicts successful information seeking more reliably than generic awareness of underspecification. This relationship is strongest in Logic-Q-MT and directionally consistent, though less clean, in GeneReg-MT.

  • Feature extraction: MSS relative mention rate R_i counts MSS variables named in strict query-planning contexts, divided by queryable variables named in those contexts, with R_i = 0 when the denominator is zero.Generic awareness of underspecification is separately coded when models say more information is needed without naming a variable.
  • Logic-Q-MT results: In Logic-Q-MT, MSS relative mention rate predicts final sufficiency (β_z = 0.528, p = 6.71 × 10−45), while underspecification awareness is near zero (β_z = 0.005, p = 0.853).The same pattern holds for second-turn hit after a first-turn miss: MSS relative mention rate remains positive (β_z = 0.365, p = 7.99 × 10−23), whereas awareness remains near zero (β_z = 0.010, p = 0.774).
  • GeneReg-MT results: GeneReg-MT shows positive MSS relative mention effects for final sufficiency in steady-state (β_z = 0.340, p = 5.39 × 10−17) and marker tasks (β_z = 1.016, p = 3.77 × 10−27).For second-turn hit, effects are also positive in steady-state (β_z = 0.274, p = 1.52 × 10−5) and marker tasks (β_z = 0.590, p = 1.59 × 10−5), but stability diagnostics are sparser.
  • Adjustment sensitivity: Dropping problem factors changes standardized log-odds ratios only modestly, including Logic-Q-MT final sufficiency from 0.528 to 0.562 and second-turn hit from 0.365 to 0.392.GeneReg-MT changes are similarly limited: steady-state final sufficiency shifts from 0.340 to 0.334, while marker final sufficiency shifts from 1.016 to 1.007.

G.1.2 Example CoTs

Annotated first-turn CoTs from Logic-Q-MT show models reasoning about rule antecedents, forbidden queries, and minimal sufficient sets while selecting sequential queries. The examples include both successful identification of relevant attributes and cases where the first query is not itself in the MSS.

  • Logic-Q-MT examples: Qwen3-4B-Thinking targeted thoughtless by querying exuberant then straightforward, while its MSS was {straightforward, weary}.Its CoT identifies weary as part of a thoughtless rule and notes that straightforward must also be checked.
  • Logic-Q-MT examples: Qwen3-30B-A3B-Thinking-FP8 targeted smart with proud followed by plain, reasoning that both antecedents of the direct rule were needed.The canonical MSS additionally contained long: {long, plain, thoughtless}.
  • Logic-Q-MT examples: Qwen3.5-122B-A10B-FP8 targeted gifted with talented followed by mean, while its MSS was {careless, mean}.The CoT considers an allowed careless query and explains that its outcome determines whether gifted is false or further antecedents are needed.
  • Logic-Q-MT examples: gpt-oss-20B-high targeted elegant with versatile followed by busy, identifying busy, sincere, and uptight as its MSS and excluding forbidden elated.Its reasoning states that elegant requires all three antecedents of the relevant rule, but elated cannot be queried.

G.1.3 Additional results · G.1.4 Prompt sensitivity · G.1.5 Preventing premature answers

The additional analyses examine interaction caps, prompt sensitivity, and whether explicit rejection of premature answers improves information seeking. Prompt effects are generally small and model-dependent, while force-asking substantially benefits some weaker models but primarily reveals elicitable rather than spontaneous capability.

  • G.1.3 Additional results: 10 turns caps Logic-Q-MT interactions in the no-budget setting, and Figure 22 reports the proportion of interactions reaching that cap.Appendix G.1.5 additionally tests rejecting answer attempts issued before k queries.
  • G.1.4 Prompt sensitivity: Prompt sensitivity is evaluated in Logic-Q-MT and GeneReg-MT by comparing default prompts with rule-order, query-guidance, and insufficiency-reminder variants.The study evaluates Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking-FP8, gpt-oss-20B-high, and GPT-5-mini.
  • G.1.4 Prompt sensitivity: Meaning-preserving rule reordering has no measurable effect in either domain, with every result remaining within run-to-run noise.Noise is calibrated by rerunning the original prompt under a different random seed and testing whether variants shift final sufficiency more than that rerun.
  • G.1.4 Prompt sensitivity: −0.040 in Logic-Q-MT and +0.002 in GeneReg-MT are GPT-5-mini’s changes under the query-guidance variant, whose effects are otherwise small and model-dependent.The strongest model is essentially unaffected, while effects concentrate in smaller open-weight models.
  • G.1.4 Prompt sensitivity: The default uses no stated budget because explicit turn budgets can change stopping behavior, while prompts were fixed before model comparison without model-specific or test-instance tuning.The k-prediction and MSS-identification prompts were minimally adapted from QuestBench to support multiple jointly necessary variables.
  • G.1.5 Preventing premature answers: The force-ask intervention leaves query selection unconstrained and permits stopping after k queries, so its gains may reflect explicit correction rather than spontaneous calibration.The protocol arguably leaks the degree of underspecification and enforces continued querying atypically.
  • G.1.5 Preventing premature answers: +9.7 pp for Qwen3-4B-Thinking and +25.7 pp for Qwen3-30B-A3B-Thinking-FP8 are the significant final-sufficiency gains from rejecting answers before k queries at k = 3.The effects for gpt-oss-20B-high and GPT-5-mini are insignificant; the intervention also increases turn counts.

G.1.6 Sensitivity to the designated MSS · G.2 GSME-Q-MT

The designated MSS has limited effect on Logic-Q-MT outcomes, with most alternative-draw differences within run-to-run noise, while alternative draws are slightly easier by construction. GSME results show that strong single-turn performance does not ensure reliable multi-turn information seeking, especially in the extended setting.

  • G.1.6 Sensitivity to the designated MSS: 96.8% of Logic-Q-MT problems have multiple MSSs, averaging 5.98, 8.74, and 10.34 MSSs at k = 1, 2, and 3.GeneReg-MT, GSME-Q-MT, and ClinGuide-MT have unique MSSs by construction; final sufficiency is invariant to MSS choice.
  • G.1.6 Sensitivity to the designated MSS: 89.7% of Logic-Q-MT problems received three pairwise-different canonical MSS draws in the resampling design.The study compared two alternative draws with the original and used different-seed reruns to calibrate run-to-run noise.
  • G.1.6 Sensitivity to the designated MSS: Final sufficiency increased by up to 7.2 percentage points at k = 3, with positive shifts in 7 of 8 comparisons and slightly fewer turns.Most differences fell within run-to-run noise.
  • G.1.6 Sensitivity to the designated MSS: Only Qwen3-4B-Thinking’s altgt2 turn reduction survived Holm correction; two further comparisons reached uncorrected p < 0.05.The additional uncorrected comparisons were Qwen3-4B-Thinking’s altgt1 turn reduction and gpt-oss-20B-high’s altgt1 final-sufficiency shift.
  • G.1.6 Sensitivity to the designated MSS: At k = 3, all original canonical MSSs were forward-derivable with mean depth 4.40, versus roughly 60% and mean depth 3.5 for resampled draws.Alternative MSSs could be sufficient by refutation, making them shallower and easier to identify; depth is defined only for forward-derivable draws.
  • G.2 GSME-Q-MT: Several models perform well when missing information is requested in one turn but become less reliable when the same information must be acquired interactively.The single-turn versus multi-turn gap is small on GSME-Q-MT, where instances are relatively simple and many models remain near saturation.
  • G.2 GSME-Q-MT: GSME-Q-MT-Ext reveals clearer differences across models and values of k than the original GSME setting.The original GSME setting is nearly saturated across models, whereas the DAG-generated complex setting shows sharp multi-turn degradation.

G.3 ClinGuide-MT full results

ClinGuide-MT results show that full query coverage is necessary for high final accuracy, while correct query order and query relevance provide additional benefits. Regression analysis identifies query coverage as the strongest predictor, followed by order correctness and, especially in harder settings, query correctness.

  • Coverage and ordering: Full coverage of the minimally sufficient set achieves substantially higher final accuracy than partial coverage across nearly all models and settings.This holds across k = 3, 4 and 0, 10, 20 distractors.
  • Regression analysis: Query coverage is the strongest and most stable positive predictor of final accuracy across model and setting pairs.It is most frequently the top-ranked predictor in the regression analysis.
  • Regression analysis: Order correctness is usually the second-ranked predictor, while query correctness adds a positive association that becomes more salient in difficult settings.Query correctness captures avoiding off-branch or irrelevant queries.

G.3.1 Open-book evaluation · G.4 20Q evaluations

Open-book access to all 59 diagnostic algorithms substantially improves ClinGuide-MT final accuracy, but performance remains unsaturated and information-seeking effects persist. The favorable setup therefore mainly tests matching cases to relevant decision-tree branches rather than selecting useful information from noisy sources.

  • G.3.1 Open-book evaluation: Open-book evaluation provides all 59 diagnostic algorithms in their original wording, including the algorithm relevant to each case.This separates information-seeking from reliance on the model’s internal medical knowledge.
  • G.3.1 Open-book evaluation: Providing the full algorithm set substantially improves final accuracy for all four evaluated models at both difficulty levels.Qwen3-4B-Thinking shows the largest gains.
  • G.3.1 Open-book evaluation: Final accuracy remains far from saturated even with access to the relevant diagnostic algorithm.The algorithm alone does not solve the task.
  • G.3.1 Open-book evaluation: Full MSS coverage yields the highest accuracy, and following the diagnostic algorithm’s suggested query order further improves performance.These coverage and ordering effects persist in the open-book setting wherever the strata are populated.
  • G.3.1 Open-book evaluation: The open-book condition is a favorable near-upper-bound setting because the 59 trees use the exact task-construction wording and clear procedural structure.Solving largely reduces to matching a case to the right tree and node and following the branch.
  • G.3.1 Open-book evaluation: The evaluation still requires models to locate the relevant decision tree among all provided algorithms.The setup is easier than deciding relevance within a large or noisy knowledge source.

G.4.1 Full results · G.4.2 Case study

In 20Q tasks, more informative questions generally improve accuracy, while multi-turn interaction steadily narrows uncertainty but yields diminishing returns. Case studies show that failures arise when models do not consolidate partial progress into reliable final disambiguation.

  • G.4.1 Full results: Higher-quality questions generally correspond to higher final accuracy, although offline question informativeness alone does not determine success.Figure 42 compares final guessing accuracy with average offline question-quality scores across datasets and examiners.
  • G.4.1 Full results: This accuracy–informativeness trend is especially clear under the GPT-5-mini examiner, where stronger guessers obtain both high question quality and high final accuracy.
  • G.4.1 Full results: Entropy decreases steadily across datasets, examiners, and almost all guesser models, confirming that generated questions progressively narrow the candidate space.Figure 43 tracks average posterior uncertainty after each turn for each guesser model, dataset, and examiner.
  • G.4.1 Full results: Later turns become less effective because the largest entropy reductions usually occur early, when broad categorical questions separate large candidate groups.The supplied passage describes a clear slowdown in most later-turn curves.
  • G.4.1 Full results: Pass mass measures posterior support for candidates whose current question would receive a pass response because it is ambiguous or not cleanly answerable by yes or no.Figure 44 reports pass mass across turns in the 20Q tasks.
  • G.4.2 Case study: The case study jointly visualizes QA traces, remaining candidate entropy, and answer-pass support to analyze why seemingly high-quality questions can still produce low final accuracy.These elements are examined for representative failure interactions.
  • G.4.2 Case study: Four representative examples illustrate weak final disambiguation, ambiguity accumulation, inconsistent belief-state updates, and searching at the wrong abstraction level.Despite making progress narrowing candidates, models often fail to consolidate it into a correct final answer.
Loading 2608.14808v1…