PaperlayerView exact revision on arXiv

Source-linked AI summary

Rational Clarification by Assistive Agents via Value-of-Information Reasoning

T. Duy Nguyen-Hien, Yee Whye Teh, Wee Sun Lee, Tan Zhi-Xuan

arXiv:2609.37588v1cs.AIcs.CLcs.HCcs.MA

TL;DR

Assistive agents must decide whether to act on ambiguous requests or ask, while prior multi-turn clarification work neglects learning from post-action corrections. REVOIR uses value-of-information reasoning to choose whether to ask or act; across two tasks it achieves a better accuracy-efficiency frontier, with ADAPT preference satisfaction exceeding a fine-tuned policy by 13–15%.

  • Problem

    Assistive agents must choose between acting on ambiguous requests and asking, while prior multi-turn clarification work neglects learning from post-action corrections.

    Source: §1 · p. 1

  • Method

    REVOIR estimates a question’s value as the expected task-reward improvement from its answer, then uses that estimate to decide whether to ask or act.

    Source: §1 · p. 2

  • Results

    Across ambiguous QA and household planning, REVOIR achieves a better accuracy-efficiency frontier; on ADAPT, preference satisfaction exceeds the fine-tuned clarification policy by 13–15%.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    In settings where users can correct actions afterward, REVOIR adapts to recognize that asking is not always efficient.

    Source: §1 · pp. 1–2

    The setup assumes users delegate all external actions to the agent and gives it an internal user model that differs from the external simulator.

    Source: §2 · p. 4

Abstract

from arXiv · show

Users of language-based assistive agents often make ambiguous requests. In response, an assistant can either directly act on its interpretation of the request --- risking misalignment with the user --- or ask a clarifying question. Which option is the most safe and helpful? A common approach is to ask questions that minimize uncertainty about the user's intent until a threshold is reached. However, this neglects the impact of uncertainty reduction on downstream performance, the costs of asking versus acting immediately, and the possibility that users may provide corrections without being asked. To navigate these trade-offs, we introduce Rational Enquiry via Value-of-Information Reasoning (REVOIR). REVOIR makes clarification decisions via inference-time reasoning about the value-of-information of a question, which captures the expected improvement in task reward due to the answer received. In two assistive tasks --- ambiguous question answering (CondAmbigQA) and preference-aligned household task planning (ADAPT) --- we show that REVOIR achieves greater success with fewer questions than approaches based on prompting, chain-of-thought, fine-tuning, or information gain, improving preference satisfaction on ADAPT by 13-15% over a fine-tuned clarification policy while requiring no training and asking five times fewer questions. Furthermore, when the assistant can receive cheap user corrections after acting, REVOIR naturally infers that asking questions is not always efficient, demonstrating the adaptivity of our approach. In contrast, we find that vanilla reasoning agents fail to adaptively clarify user requests, and request fewer clarifications as reasoning effort increases.

1 Introduction

REVOIR treats clarification as a decision under uncertainty: an assistant should weigh the value of an answer against the costs of asking and the possibility of later correction. This addresses the risk of misinterpreting ambiguous requests while recognizing that asking is not always necessary.

1 Introduction

  • REVOIR formulates clarification as a cooperative game and estimates each question’s value by its expected improvement in task reward, then chooses whether to ask or act.It updates beliefs about user intentions in natural language and accounts for learning through both answers to questions and possible corrections after acting.Source: §1 · pp. 1–2
  • On ADAPT, REVOIR achieved 56–59% preference satisfaction versus 44% for the fine-tuned clarification policy, a 13–15% gain, with roughly 5× fewer questions and around 6% above always ask.Across CondAmbigQA and ADAPT, the paper reports greater task success and preference alignment with fewer questions than prompting, generic reasoning, fine-tuning, and expected-information-gain baselines.Source: §1 · p. 2
  • With post-hoc corrections, REVOIR infers that asking often adds no benefit over acting; generic reasoners underperform it and generally ask fewer clarifications as reasoning effort rises.The correction setting is a variant of the question-answering task in which users can correct the assistant after it acts.Source: §1 · p. 2
  • Prior clarification fine-tuning often assumes fixed question costs and terminal commitments, while expected information gain ignores variable costs and downstream reward; REVOIR handles costs and task value.Expected information gain can favor questions that reduce uncertainty without supplying information relevant to task performance.Source: §1 · pp. 2–3

2 Clarificatory Assistance as a Cooperative Game

The section adapts assistance games to language-based interaction, modeling assistance under hidden user intent and allowing either the assistant or user to control termination.

2 Clarificatory Assistance as a Cooperative Game

  • The agent initially lacks the user’s intent, which determines the goal reward received in terminal states.The intent is not fully revealed by the initial request.Source: §2 · p. 3
  • Unlike prior clarification tasks, goal rewards can vary continuously, so maximizing reward need not require fully discovering the user’s intent.The shared reward also includes user-configurable costs for questions, answers, and wrong assistive actions.Source: §2 · p. 3
  • The game lets either the assistant or user control termination, allowing users to correct an action or end the conversation afterward.Prior clarification formulations generally end interaction once the assistant acts (Andukuri et al., 2024).Source: §2 · pp. 3–4
  • Fixing a user policy reduces the assistance game to an assistive partially observable Markov decision process, whose solution is optimal relative to that model.Experiments simulate users with an LLM given the true or hypothesized intent, while the assistant’s internal user model differs from the simulator; delegated external actions are assumed.Source: §2 · pp. 3–4

3 Rational Enquiry via Value-of-Information Reasoning

REVOIR is a limited-lookahead hierarchical policy for choosing whether to ask or act, replacing intractable exact POMDP planning with value-of-information estimates. It models intent uncertainty with weighted LLM-generated particles and compares the cost-adjusted expected value of asking or acting.

3.1 Belief Updating over User Intent

  • To represent intent uncertainty, REVOIR approximates the intractable posterior with weighted particles whose hypotheses are proposed and semantically scored by LLMs.This avoids requiring an enumerable intent space or trustworthy prior and user-likelihood models; LLM log-probabilities can be sensitive to irrelevant features such as typos and text length.Source: §3.1 · p. 4

3.2 Evaluating the Value of Information (VoI)

  • REVOIR estimates question value by simulating answers, updating beliefs, and evaluating the expected reward of acting; when corrections are possible, it also values information gained by acting.An LLM-based reward model scores actions against intent hypotheses, and best-of-K sampling selects candidate questions and actions. Action value includes expected future reward after a user correction, or just immediate reward when actions are terminal.Source: §3.2 · pp. 5–6

3.3 Deciding Between Asking and Acting

  • REVOIR asks only when the cost-adjusted value of asking exceeds the value of acting immediately; otherwise, it acts.The expected values are estimated by sampling, using the best question or action among the candidates.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate REVOIR on ambiguous question answering (CondAmbigQA) and preference-aligned household planning (ADAPT). Across both tasks, results show benefits from value-of-information-based clarification, including adapting clarification effort to user corrections and task trade-offs.

4.1.1 Experiment Results

  • Under agent termination, REVOIR beats all ReAct variants in correctness; under user termination, it beats all baselines while using the fewest clarifications among non-ReAct methods.REIGN and Entropy Thresholding are close in correctness, with REIGN slightly ahead, but use 1.3–2.8× more questions than REVOIR with GPT-5.4-mini; ReAct variants clarify little at substantial correctness cost.Source: Figure 2 · p. 7, §4.1.1 · p. 7
  • Increasing GPT-5.4-mini ReAct reasoning effort yields non-monotonic, limited gains and very few clarifications above low effort; more SC-BoN-ReAct samples also do not consistently improve results.SC-BoN-ReAct exceeds medium-to-high-budget REVOIR only with up to 2–4× more tokens, suggesting generic reasoning effort and parallel sampling do not substitute for principled value-of-information reasoning.Source: Figure 2 · p. 7, §4.1.1 · p. 8
  • REVOIR outperforms baselines for most trade-off ratios (α > 0.008 with agent termination; α > 0.002 with user termination) and ranks second at α = 0.The comparison uses the best configuration for each method at each α, with utility measured as AnswerCorrectness − α · Effort.Source: §4.1.1 · p. 7, Figure 3 · p. 8
  • When user corrections are available, only REVOIR reduces total clarifications by asking fewer questions; all other methods increase both questions and clarifications.The comparison is between agent-termination and user-termination variants on Llama-3.1-8B.Source: Figure 4 · p. 8, §4.1.1 · p. 8

4.2.1 Experiment Results

  • 58.8% seen-persona and 56.0% unseen-persona PSR with ICL let REVOIR outperform ReflectionDPO (44%) by 13–15 points using 5× fewer questions, and outperform all finetuning and prompting baselines.In zero-shot, REVOIR achieved 51.5% PSR on seen personas and 49.5% on unseen personas, virtually matching Always-Ask with 11× fewer questions. REVOIR was evaluated in a two-stage ADAPT variant that asks all questions before physical planning, unlike baselines that interleave asking and acting.Source: Figure 5 · p. 9, §4.2.1 · p. 9, §4.2 · p. 9

5 Conclusion · APPENDIX

REVOIR improves the accuracy-efficiency frontier for ambiguous question answering and preference-aligned planning, but its one-step estimates and limited interaction design leave clear directions for future work. The paper also reports how generative AI was used and reviewed.

5 Conclusion

  • Across ambiguous question answering and preference-aligned planning, REVOIR achieves a better accuracy-efficiency frontier, adapts to user corrections, and surpasses strong fine-tuning baselines while asking fewer questions.The conclusion describes these gains without providing numerical values here.Source: §5 · p. 10
  • REVOIR’s tractable one-step value-of-information estimate is myopic, motivating efficient multi-step estimates.Source: §5 · p. 10
  • REVOIR is not designed for dual control or fully interleaved questions and non-terminal actions, which will require efficient value-of-information estimation under interleaving.Source: §5 · p. 10
  • Although REVOIR is reasonably robust to imperfect user modeling, evaluation with real users and their idiosyncrasies remains an important deployment step.Source: §5 · p. 10

AI use statement

  • Generative AI tools were used to augment a dataset, implement parts of the methods and result visualization and interpretation, and suggest manuscript wording.They were not used to generate synthetic datasets, develop theoretical models or conceptual frameworks, propose or refine hypotheses, design or provide feedback on methodology or experiments, translate, or clean datasets.Source: AI use statement · p. 10
  • The authors reviewed all AI-assisted work and accept responsibility for the final content, including AI-assisted text, claims, and artifacts.AI assistance in mathematical claims was minimal and limited to extending known results with elementary techniques, formatting, and restatement.Source: AI use statement · p. 10

A Value of Adaptivity · B Recovering Expected Information Gain · C Extended Details on CondAmbigQA

Clarification can improve expected reward when the agent adapts its action to the answer, while a non-adaptive policy gains no benefit from more information. Expected information gain is a special case of REVOIR when reward measures only uncertainty reduction and asking is cost-free.

A Value of Adaptivity

  • Under terminal, information-only clarification with a fixed action set, adapting the chosen action to the answer cannot reduce expected reward and can improve it when answers favor different actions.The result assumes the feasible actions and each fixed action’s utility remain unchanged by the question.Source: A Value of Adaptivity · p. 14
  • A policy that chooses the same action before and after clarification gains no expected reward from the additional information.By the law of total expectation, the fixed action’s expected reward after updating equals its reward under the current belief.Source: A Value of Adaptivity · p. 14
  • With belief-conditioned candidate actions, best-of-K sampling after clarification can outperform exact maximization before it when the answer’s expected benefit exceeds sampling error.The candidate distribution may change when the belief updates.Source: A Value of Adaptivity · p. 14
  • Under the stated sampling assumptions, the expected shortfall from the optimal post-answer action is at most ε + (D − ε)δ^K.This bound uses K independent draws, a reward interval of width D, and probability at least 1 − δ that each draw is within ε of optimal.Source: A Value of Adaptivity · p. 14
  • As K grows, δ^K approaches zero, so clarification can outperform pre-clarification exact maximization when expected optimal-reward gain exceeds ε; for K = 1, it must exceed ε + (D − ε)δ.The passage states these thresholds for the sampling guarantee’s assumptions.Source: A Value of Adaptivity · p. 14

B Recovering Expected Information Gain

  • Expected information gain is a special case of REVOIR when goal reward is log belief probability, costs are zero, and acting provides no information.Under these conditions, the value difference between asking and acting is H(b) − H(b | mask).Source: B Recovering Expected Information Gain · p. 15

C.1 Benchmark Configuration

The benchmark configuration uses CondAmbigQA, simulated users, and an LLM-judged answer-correctness metric. The metric is checked against oracle and direct-answer anchors, which it ranks as expected.

Dataset

  • CondAmbigQA contains 2,000 ambiguous queries with explicit contextual conditions representing valid intents and Wikipedia-supported ground-truth answers.The queries are drawn from AmbigNQ; answers were generated through human–LLM collaborative annotation.Source: §C.1 · p. 15
  • Each question-condition pair receives a GPT-4o-generated disambiguating question, and question-level splits prevent conditions for training questions from appearing in evaluation.The questions also guide the user simulator and serve as oracle reflection questions for ReflectionDPO (Appendix H); split statistics appear in Table C.1.Source: §C.1 · p. 15, Table C.1 · p. 15

User Simulator

  • The user simulator uses GPT-4o-mini and the ground-truth intent to answer clarification requests or dismiss unanswerable inquiries.In the user-termination setting, it may accept the assistant’s answer or correct it; REVOIR instead relies on a proxy user model with a different LLM backbone.Source: §C.1 · p. 15

Evaluation Metric

  • AnswerCorrectness uses contradiction, omission, relevance, and acumen checks to judge the final answer against the true-intent answer, following Li et al. (2025).The G-Eval rubric adds an acumen reward for relevant answers with greater depth and detail, since assistants lack web-retrieved resources.Source: §C.1 · p. 15
  • 0.995 out of 1 for Oracle, 0.619 for Llama-3.1-8B and 0.63 for GPT-5.4-mini Oracle-ReAct, and around 0.50–0.56 for Direct Answer anchor the metric’s expected ranking.Interactive methods score between the topline and baseline anchors.Source: §C.1 · pp. 15–16

C.2 REVOIR Implementation in CondAmbigQA · C.3 Baselines for CondAmbigQA · D Additional Results on CondAmbigQA

The CondAmbigQA implementation operationalizes REVOIR with belief-based, cost-adjusted VoI question selection and an adaptive answer policy, alongside comparisons with information-gain, prompting, reasoning, and fine-tuned baselines.

C.2 REVOIR Implementation in CondAmbigQA

  • Message costs are zero within user-specified word budgets and rise to at most one while messages remain below twice their respective budgets.The implementation assumes costs are mostly borne by users and token-inference costs are marginal.Source: §C.2 · p. 16
  • REVOIR represents intent with five sampled hypotheses, scores their consistency with conversation history, and normalizes the weights anew at every turn.A proposer may retain or prune hypotheses from the previous belief; the scorer assigns consistency scores that are normalized to sum to one.Source: §C.2 · p. 16
  • Its internal user model forecasts answers to questions and corrections, but differs from the simulator in access to intent, LLM, and prompts.The internal model lacks access to the true intent and uses a different LLM and prompts from the user simulator.Source: §C.2 · p. 16
  • REVOIR selects among five generated questions by cost-adjusted VoI, estimated by simulating one user answer for each hypothesis in its current belief.A question generator proposes the five candidates from the particle belief, and the VoI estimate averages the cost-adjusted expectand across simulated answers.Source: §C.2 · p. 16
  • Even with one answer candidate, REVOIR can remain belief-adaptive: updated beliefs can yield better answers, so selecting among multiple candidates is not required for clarification to improve downstream decisions.The implementation samples K = 1 answer candidate; the stated sufficient condition for positive value of information can hold at K = 1.Source: §C.2 · pp. 16–17
  • For acting, REVOIR averages goal reward over its belief and approximates correction probability as 1−max_n w_n, increasing expected corrections under greater uncertainty.The implementation uses one answer candidate; the correction-probability approximation is fixed across intents and answers and depends on the peak belief weight.Source: §C.2 · pp. 16–17

C.3 Baselines for CondAmbigQA

  • CondAmbigQA baselines include cost-aware entropy, direct answering, reasoning, threshold, self-consistency, fine-tuned, and privileged-answer approaches.REIGN uses normalized negative belief entropy as a cost-aware EIG comparator; Direct Answer skips clarification, while entropy thresholding asks until belief concentration crosses a tuned threshold. ReAct interleaves reasoning and actions without explicit belief, principled stopping, or cost accounting (Yao et al. (2022)); SC-BoN-ReAct uses majority-voted samples (Wang et al. (2023)), ReflectionDPO-ReAct learns via privileged-teacher reflection (Patel et al. (2025)), and Oracle-ReAct receives the privileged answer.Source: §C.3 · pp. 17–18

D.1 Full Results of Llama-3.1-8B on CondAmbigQA

Across CondAmbigQA results, REVOIR balances answer correctness against clarification costs and adapts its behavior when users can correct answers. It also achieves comparable or better correctness than inference-time scaling at similar token budgets.

D.1 Full Results of Llama-3.1-8B on CondAmbigQA

  • Under agent termination, REVOIR reaches 0.55 correctness in 5.6 questions, while REIGN asks up to 9 and plateaus below 0.54; REVOIR’s advantage is robust to dataset noise.REVOIR Pareto-dominates REIGN at matched word budgets, while Entropy Thresholding asks about 4 questions and achieves the lowest correctness among multi-turn methods.Source: §D.1 · p. 18, Figure D.1 · p. 19
  • With user corrections, REVOIR yields 0.56–0.57 correctness using 2–4 total clarifications; REIGN reaches 0.58 at 6.9, while Entropy Thresholding uses 12 and is less correct.REVOIR uses substantially fewer words than its budget, relying on user corrections rather than asking more questions.Source: §D.1 · p. 18, Figure D.2 · p. 20
  • At comparable token budgets, REVOIR achieves similar or greater correctness than SC-BoN-ReAct, whose cost reaches ∼74K tokens at N=20—over 3× the most expensive REVOIR operating point.SC-BoN-ReAct’s token cost grows linearly with N, while its correctness gains are marginal and non-monotonic.Source: Table D.3 · p. 21

D.2 Gemini 3.5 Flash-Lite and Claude Haiku 4.5 on CondAmbigQA

REVOIR generally led the interactive methods on AnswerCorrectness, while Gemini 3.5 Flash-Lite and Claude Haiku 4.5 showed different behaviors in direct answering and clarification. The evaluation setup also constrained which methods and reasoning levels could be tested.

D.2 Gemini 3.5 Flash-Lite and Claude Haiku 4.5 on CondAmbigQA

  • 0.70 (Gemini 3.5 Flash-Lite) and 0.76 (Claude Haiku 4.5) were their best scores over GPT-5.4-mini non-interactive toplines; REVOIR generally led interactive methods in AnswerCorrectness with minimal/no reasoning.These model-specific results appear alongside the finding that REVOIR generally achieved the highest AnswerCorrectness among interactive methods.Source: Figure D.3 · p. 21, §D.2 · p. 22
  • The replication omitted Entropy Thresholding for cost reasons; Gemini was tested with minimal reasoning for REVOIR and REIGN, and medium reasoning for SC-BoN-ReAct, while Claude thinking levels used token budgets from 1,024 to 12,288.Gemini's reasoning could not be disabled, and lower reasoning levels for SC-BoN-ReAct produced empty responses due to context-window overflow.Source: §D.2 · pp. 21–22
  • Gemini 3.5 Flash-Lite DirectAnswer baselines with non-minimal reasoning also outperformed the evaluated interactive methods, aided by verbose answers covering multiple possible intents.The authors describe these answers as shotgun-style and show average one-shot query word counts across reasoning efforts in Figure D.4.Source: §D.2 · p. 22, Figure D.4 · p. 22
  • The evaluation did not penalize answer length or intent-irrelevance when answers stayed on-topic and covered the target intent, benefiting Gemini's shotgun approach.The authors note that such hedging may suit users prepared for an influx of information, such as in a search engine, but may not always be appropriate.Source: §D.2 · pp. 22–23
  • Claude Haiku 4.5 asked very few questions across interactive methods and settings, and nearly stopped asking when user corrections were anticipated, while REVOIR remained the best-performing interactive method.REVOIR's performance advantage is attributed to reasoning about the value of information associated with each action.Source: §D.2 · p. 23

D.3 Results Under Inattentive User Simulator · E Validating the LLM-simulated user and LLM judge with human annotations

REVOIR’s CondAmbigQA advantage persists with inattentive users, while the human-annotation pilot assesses conversation termination and final-answer correctness. The supplied pilot passages describe its design but do not report annotation outcomes.

D.3 Results Under Inattentive User Simulator

  • Under inattentive users, REVOIR leads in agent-termination correctness for both backbones; under user termination, it matches REIGN within 0.004 and asks half as many questions on 8B.On 70B under user termination, REVOIR has the highest correctness and fewest total user interventions.Source: Figure D.5 · p. 23, §D.3 · p. 23
  • The inattentive simulator gave dismissive responses with probability 0.2 per clarification turn, creating a mismatch with REVOIR’s internally modeled user.Due to compute and cost constraints, the replication covered Section 4.1 for Llama-3.1-8B and 70B with planning budgets of 100 assistant words and 50 user words; AnswerCorrectness was independent of dialogue length and cost.Source: §D.3 · p. 23
  • Across dismissal probabilities 0.3–0.9, only REVOIR and REIGN are robust to dismissive noise; ReAct baselines’ AnswerCorrectness visibly declines as dismissal increases.The additional sweep used Llama-3.1-8B; the reported pattern extends the findings under the 0.2 dismissal setting.Source: §D.3 · p. 23, Figure D.6 · p. 24

E Validating the LLM-simulated user and LLM judge with human annotations

  • The pilot conversation-replay study assessed human judgments of user-simulator termination decisions and final-answer AnswerCorrectness ratings.Annotators read logged GPT-5.4-mini conversations from REVOIR and the strongest baseline, ReAct (R/E: High), in the user-termination setting.Source: E Validating the LLM-simulated user… · p. 24
  • Nine university members annotated eight distinct item sets, with one response dropped because an item set was erroneously duplicated between annotators.Source: E Validating the LLM-simulated user… · p. 24
  • Reported intervals are two-sided 95% Wilson score intervals for binomial proportions, chosen because they remain well behaved with modest samples and rates near zero or one.Source: E Validating the LLM-simulated user… · p. 24

E.1 Does the user simulator terminate where a person would? … G.5 SC-BoN-ReAct Parameters •

This appendix documents the CondAmbigQA evaluation framework and experimental setup, including checks of simulated-user termination and correctness judging, episode and scoring details, and prompts, procedures, and configurations for value estimation, answer generation, and clarification methods.

E.1 Does the user simulator terminate where a person would?

  • Human-simulator agreement was 72/89 = 0.809 overall and 64/71 = 0.901 on simulator-accepted episodes.Agreement was 26/29 = 0.897 after corrections and 38/42 = 0.905 for first-answer acceptance. Disagreements were asymmetric: the simulator rejected 10 answers accepted by humans, while humans rejected 7/15 simulator-accepted answers; it concurred on only 8/18 = 0.444 simulator rejections.Source: Table E.1 · p. 25, §E.1 · p. 25

E.2 Does the LLM judge track human answer-quality judgments?

  • On non-tied comparisons, human preferences and AnswerCorrectness agreed in 41/57 = 0.719 cases.Annotators marked 39/96 comparisons equally good; among the 57 with an expressed preference, agreement with AnswerCorrectness was 41 cases.Source: Table E.2 · p. 26, §E.2 · p. 26

F CondAmbigQA Prompt Library and Episode Lifecycle

  • The prompt library presents verbatim CondAmbigQA templates with run-time values substituted by placeholders, alongside episode-lifecycle and user-simulator descriptions.This section introduces the materials covered in the following subsections.Source: F CondAmbigQA Prompt Library and… · p. 26

F.1 Episode Lifecycle

  • Each episode tracks a particle belief over interpretations, chooses whether to ask or answer using cost-adjusted VoI, and scores the final answer with AnswerCorrectness.A hard budget limits clarifying questions; under user termination, the simulator may issue a correction after the agent answers.Source: §F.1 · p. 26

F.2 User Simulator

  • The simulator answers clarifying questions from the ground-truth condition, judges whether an answer matches intent, and may issue a steering correction if unsatisfied and budget remains.Questions that cannot be answered from the condition receive a dismissal; the correction points toward the intended interpretation without revealing the answer.Source: §F.2 · pp. 26–27

F.3 Evaluation Judge and Planning-Time Reward Proxies

  • CondAmbigQA’s final reward is a rubric-based G-Eval AnswerCorrectness score, while the planning-time proxy uses verbalized scoring normalized from 0–10 to [0, 1].The final rubric includes contradiction, omission, relevance, and acumen criteria; the proxy mirrors it but drops the contradiction check because no reference answer is available at planning time.Source: §F.3 · p. 28

F.4 REVOIR and REIGN Prompt Templates

  • REVOIR, REIGN, and Entropy Thresholding share prompt templates but differ in their decision-rule reward; the hypothesis proposer generates interpretation hypotheses and the consistency scorer weights them against user responses.The proposer refines likely hypotheses and prunes unlikely ones; consistency scores use a 0–10 scale.Source: §F.4 · p. 29
  • REVOIR and REIGN templates forecast hypothetical user answers or corrections, evaluate candidate questions with one-step lookahead, and value acting by expected reward over belief hypotheses.The question generator proposes K distinct questions; the answer generator produces one response conditioned on the full belief summary.Source: §F.4 · pp. 30–31

F.5 ReAct and SC-BoN-ReAct

  • ReAct alternates one-sentence reasoning with ask-or-answer actions without an explicit belief; under user termination, anticipated corrections encourage earlier commitment and updating from feedback.The dialogue scaffolding injects user answers as observations and, in user-termination mode, corrections as clarification observations.Source: §F.5 · pp. 32–33
  • SC-BoN-ReAct samples K ReAct completions, uses a self-consistency vote to choose asking versus answering, then an LLM judge to select among candidates of the winning type.The selection judges choose the best question or answer from same-type candidates given the original question and dialogue history.Source: §F.5 · p. 34

F.6 ReflectionDPO-ReAct

  • ReflectionDPO-ReAct uses a fine-tuned model, while privileged reflection with the ground-truth answer generates Answer-type training signal and serves as the Oracle-ReAct teacher at inference.The reflection template asks which Answer action should have been produced for the conversation.Source: §F.6 · p. 35
  • ReflectionDPO-ReAct’s reflection prompt asks the model to output exactly two lines: a thought and an Answer action aligned with the true extractive answer.The prompt supplies the correct document’s extractive answer as the target.Source: §F.6 · p. 35

F.7 Non-Interactive Baselines

  • Direct Answer responds in one turn under full ambiguity, while Oracle-ReAct rephrases the ground-truth answer and Oracle reference submits it verbatim.Direct Answer uses a minimal system prompt; Oracle-ReAct receives the answer at every call, and Oracle reference is the metric ceiling.Source: §F.7 · p. 35

G.1 Environment Roles

  • REVOIR’s reasoning components could each use a different model, but experiments use a unified backbone to isolate method effects; implementation configures agent, user-simulator, and post-episode judge roles.Source: §G.1 · pp. 35–36

G.2 Global Episode Parameters

  • Across interactive methods, each episode allows 10 clarifying questions; expert-model generation uses temperature 0.0 (greedy), while macro-action selection greedily chooses between asking and acting.Under user termination, users issue at most five corrections after unsatisfactory answers; main REVOIR and REIGN runs use word budgets of 100 agent words and 50 user words. Agent generation uses agent_max_tokens 2048; user generation uses user_max_tokens 128.Source: §G.2 · p. 36

G.3 VoI Planner Parameters (REVOIR, REIGN, Entropy Thresholding) •

  • REVOIR, REIGN, and Entropy Thresholding use five belief hypotheses and evaluate five candidate questions per turn, with a single belief-conditioned answer for acting.Entropy Thresholding’s normalized concentration threshold is tuned on the Dev split for each agent model and termination variant.Source: §G.3 · p. 36, Table G.1

G.5 SC-BoN-ReAct Parameters •

  • SC-BoN-ReAct uses five self-consistency samples in main comparisons and tunes its answer-vote threshold on Dev for each model and termination variant.For Llama-3.1-8B, ensemble size is additionally swept from 5 to 20 and the threshold is tuned per K; sampling temperature is 0.7, with greedy best-of-N selection.Source: §G.5 · p. 37, Table G.2 · p. 37

H Adapting ReflectionDPO to Ambiguous Question Answering … I.2 REVOIR Variants in ADAPT

The sections detail a task-specific ReflectionDPO adaptation for ambiguous question answering and how REVOIR is adapted and implemented for ADAPT. They describe the training signals and limitations, two-stage execution, and choices for belief tracking and reward estimation.

H Adapting ReflectionDPO to Ambiguous Question Answering

  • ReflectionDPO uses student rollouts and oracle answers to derive reflection questions, then prefers either a useful question or a fluent rephrasing of the oracle answer.The oracle provides ground-truth clarification questions and extractive answers; an Oracle-ReAct model rephrases answers to avoid aligning directly to verbose oracle text.Source: H Adapting ReflectionDPO to Ambiguous… · p. 37
  • Ask pairs require oracle-answer gains above ε1 (at least 0.05 nats/token), while Answer pairs require rollout disadvantage below ε2; median-based step stratification shifts ask-heavy ratios 3.6:1→3.8:1 and ask-moderate ratios 1.5:1→1.7:1.Source: H Adapting ReflectionDPO to Ambiguous… · pp. 37–39
  • Because long-form DPO can reduce both responses’ likelihoods without improving alignment, the adaptation uses ORPO and anchors the chosen response on the teacher’s fluent rephrasing.The authors reject anchoring on terse extractive answers because that could penalize fluent paraphrases.Source: H Adapting ReflectionDPO to Ambiguous… · p. 39
  • The authors speculate that the adaptation underperforms because its extractive-answer likelihood signal may not predict open-ended answer quality, while its small curated dataset may underdetermine a reliable policy.They contrast this task-specific training with REVOIR’s inference-time reasoning about task-specific value of information, which requires no training.Source: H Adapting ReflectionDPO to Ambiguous… · p. 39

I.1 Adapting ADAPT to REVOIR

  • ADAPT separates clarification from execution: REVOIR asks iteratively or commits to execution, then a policy executes a full plan using the updated belief; this limits information gathering but preserves grounded planning complexity.The two-stage design replaces ADAPT’s freely interleaved questions and physical actions.Source: §I.1 · pp. 39–40

I.2 REVOIR Variants in ADAPT

  • ADAPT variants use either joint beliefs over complete preference sets or factored Bernoulli beliefs over individual preferences, enabling targeted questions about uncertain dimensions.Joint beliefs use K = 6 particles; factored beliefs support a larger effective particle count.Source: §I.2 · p. 40
  • Belief weights are either rescored from the full history each turn or incrementally updated with the new message, with replacement particles replaying the conversation to catch up.The incremental scheme caches retained particles’ accumulated log-weights to avoid redundant recomputation.Source: §I.2 · p. 40
  • A planner, simulator, and proxy judge estimate full-plan rewards; joint beliefs aggregate particle scores, while factored beliefs weight preference contributions by their posterior inclusion probabilities.Combining the two belief representations with the two update schemes yields four REVOIR variants.Source: §I.2 · p. 40

I.3 Full Results on ADAPT … K.1 Environment Roles

This section presents ADAPT results and experimental methods, covering clarification performance, belief-conditioned question and value estimation, and preference-aware action execution. It also describes the episode and user-simulation setup, model implementations, and reward evaluation.

I.3 Full Results on ADAPT

  • 56.3–58.8% average PSR on seen personas and 55.8–56.0% on unseen let few-shot Factored REVOIR exceed Always-Ask by 6 points and Reflection-DPO (44%) by 13–15 points with ∼5× fewer questions.REVOIR requires no task-specific training; STaR-GATE fails to exceed the non-interactive Baseline LLM (34%).Source: Table I.1 · p. 41, §I.3 · pp. 41–42
  • 49.2–51.0% average PSR for zero-shot Factored-Belief variants, versus 33.0–36.4% for Joint-Belief variants, nearly matches Always-Ask LLM (50.9–52.1%) with 11× fewer questions (2 vs. 22).Factored beliefs enable targeted elicitation by tracking each preference independently, rather than scoring potentially poorly calibrated complete preference sets.Source: §I.3 · p. 41
  • Factored REVOIR shows minimal seen-to-unseen PSR degradation (58.8 →56.0% for Incremental Update with ICL), while Joint variants have 7–8% standard deviations on unseen personas.Joint-Belief REVOIR still reaches 48.0−−50.2 on unseen personas in the few-shot case.Source: §I.3 · pp. 41–42

J ADAPT Prompt Library and Episode Lifecycle

J.1 Episode Lifecycle

  • Each episode elicits preferences by comparing V ask(s, b) with V act(s, b), then executes a multi-step plan conditioned on the elicited belief; episodes are scored by PSR.Evaluation uses four benchmark splits; REVOIR runs training-free few-shot or zero-shot regimes, with seen-versus-held-out personas indicating whether test personas appeared among few-shot examples.Source: §J.1 · p. 42

J.2 User Simulator

  • The LLM user simulator answers preference questions in character, deflects questions about object location or availability, and stays consistent; ADAPT has no satisfaction check or unprompted correction.Source: §J.2 · p. 42

J.3 Hypothesis Proposers and Consistency Scorers

  • Joint belief proposes candidate preference profiles regenerated each turn and scores holistic conversation consistency; scoring uses either batch rescoring or incremental updates.Incremental updates accumulate per-turn scores, scoring retained particles on new turns and replaying history for fresh particles.Source: §J.3 · pp. 43–44
  • Factored belief proposes atomic preferences and maintains independent Bernoulli beliefs, with batch scoring or independent per-atom incremental scoring.Atoms cover the full breakfast experience, including preparation, ingredients, sides, beverages, dietary restrictions, serving style, and portion size.Source: §J.3 · pp. 45–46
  • Consistency scorers rate how strongly the conversation supports a preference on a 0–10 scale, considering all user responses and using the full range.The scoring prompt frames 10 as highly consistent, 5 as neutral or uninformative, and 0 as highly inconsistent.Source: §J.3 · p. 47

J.4 Question Generator πask

  • Candidate questions are generated from the task, scene, dialogue, and current belief, with joint and factored variants rendering that belief differently.The prompt asks for one targeted, natural question, favors open-ended preference elicitation, and avoids asking about information already provided.Source: §J.4 · p. 47

J.5 VoI Lookahead: Response Forecasting

  • To estimate question value, the agent samples hypotheses, forecasts each hypothesis’s likely answer by role-playing its user, then re-evaluates goal reward on the updated belief.The joint variant samples profiles, while the factored variant samples atom compositions.Source: §J.5 · pp. 48–49

J.6 Goal-Reward Pipeline (Planner–Simulator–Proxy Judge)

  • The goal-reward pipeline plans from the full belief, simulates the resulting scene state, and uses a proxy judge to score preference outcomes.Under factored beliefs, preference outcomes are aggregated as a mean-field weighted PSR, with each preference weighted by its inclusion probability.Source: §J.6 · pp. 49–51

J.7 Execution Policy πact

  • The execution policy chooses one action at a time from the full elicited belief, constrains generation with a live scene-specific grammar, and runs greedily until Declare Done.The grammar enforces syntactically valid actions over currently valid entities; environment observations are appended to the rollout when available.Source: §J.7 · p. 52
  • REVOIR’s execution policy selects the next task action without repeating the last, using user preferences, gathered information, and the house layout as context.Its prompt asks what action is required to achieve the task.Source: §J.7 · p. 53

K.1 Environment Roles

  • All ADAPT experiment roles use one locally served Llama-3.1-70B-Instruct model, while programmatic persona-preference predicates compute the final-state reward without a separate judge.The roles include the user simulator and REVOIR’s hypothesis proposer, consistency scorer, question generator, planner–simulator–proxy judge, and execution policy.Source: §K.1 · p. 53

K.2 Benchmark Protocol • … K.6 Sampling Temperatures

The appendix specifies ADAPT’s evaluation protocol, belief and reward configurations, elicitation and execution rules, and sampling temperatures.

K.2 Benchmark Protocol •

  • The benchmark uses 16 personas, 8 breakfast tasks, and four-fold cross-validation, reusing the original ADAPT four-way split and reporting means and standard deviations across split means.Each split has 12 seen and 4 held-out personas.Source: §K.2 · p. 53

K.3 Belief and Reward Settings

  • The four variants combine joint beliefs over 6 profile particles or factored Bernoulli beliefs over 16 atomic preferences with two update schemes.All variants share the goal-reward pipeline.Source: §K.3 · p. 53
  • Goal reward uses a planner–simulator–proxy judge pipeline, while verbalized 0–10 consistency scores update joint softmax weights or factored per-atom log-likelihood ratios.The pipeline evaluates an explicit generated plan and independently scores state deduction and each preference.Source: §K.3 · p. 53

K.4 Elicitation-Stage Parameters •

  • Each elicitation turn proposes 3 deduplicated candidate questions; VoI lookahead enumerates 6 joint profiles or samples 4 factored atom compositions per question.Candidates come from the best-of-K proposer.Source: §K.4 · p. 53
  • Elicitation has no hard question cap and stops when V act ≥ V ask − C(mask), with a flat per-question cost of 5 × 10−4 subtracted from V ask.The minimal cost avoids infinite episodes because the original baselines are neither cost-aware nor question-throttled.Source: §K.4 · pp. 53–54

K.5 Execution-Stage Parameters •

  • Execution uses greedy, grammar-constrained decoding with the grammar rebuilt from the live scene each step, and has an unlimited step budget until Declare Done.Source: §K.5 · p. 54

K.6 Sampling Temperatures

  • Sampling temperatures are 0.8 for question generation, 0.7 for VoI response forecasting, 0.3 for planner plans, and 0.0 for scoring, judging, and execution actions.Source: §K.6 · p. 54

Related papers

Loading 2609.37588v1…