CLEAT: Learning Value-Guided Clarification for Interactive Agents

1 Introduction

• Users may leave goals and constraints unstated, so effective assistance requires aligning agent decisions with intended goals through interaction.Clarification can reveal information without improving decisions; an arrival deadline can distinguish flights when airline preference cannot.Source: §1 · p. 1
• Clarification value can change during training because the policy may improve at using answers and at proceeding on its own by different amounts.Thus, information gained or a successful outcome alone does not establish that asking improved task utility.Source: §1 · p. 2
• CLEAT estimates residual query value by comparing a targeted clarification with native continuation from the same history under the current policy.The paired outcomes provide value targets for the controller and advantages for policy learning; native continuation retains the agent’s own ability to ask.Source: §1 · p. 2
• CLEAT achieves the highest mean score in all ten evaluation settings; gains over the strongest baseline reach 13.2 percentage points on UserGym Travel and 6.5 percentage points on IN3 underspecification judgment accuracy.Ablations show policy improvement without controller-based inference selection, with higher mean scores from updating rather than freezing the controller in all six ablation settings.Source: §1 · p. 2
• Predicted values guide rollout collection, while observed returns drive sampling-corrected policy updates and fresh outcomes renew value supervision as the agent learns.The framework co-trains clarification-value guidance and the task policy from shared interaction outcomes.Source: §1 · p. 2

2.1 Interactive Intent Alignment

• Interactive tasks may begin with requests that incompletely or inaccurately express the goals and constraints determining success.Clarification informs task decisions rather than serving as an objective in itself.Source: §2.1 · pp. 2–3
• The task policy acts on interaction history that only partially reveals the user’s latent goal, and asking-versus-proceeding continuations are compared from the same history and goal.Training assumes access to an environment that can sample multiple continuations under a fixed policy for each comparison.Source: §2.1 · p. 3

2.2 Group-Relative Policy Optimization

• Group Relative Policy Optimization normalizes sampled episode utilities within a group to construct advantages without a learned critic.CLEAT retains policy-ratio clipping and reference-policy regularization while modifying rollout collection and utility use.Source: §2.2 · p. 3

3 CLEAT

• CLEAT co-trains a task policy and clarification-value controller from paired rollouts, sharing observed returns as controller labels and policy advantages.The updated controller guides target selection in the next round, and new rollouts under the updated policy provide fresh value labels.Source: §3 · p. 3

3.1 Problem Setup and Residual Query Value

• Residual query value (RQV) measures the utility added by asking about a target rather than letting the current policy continue, including its own questions and interaction costs.Both continuations use the same policy after the initial action.Source: §3.1 · p. 3

3.2 Learning from Paired Outcomes

• Paired QUERY and PASS rollouts from the same history estimate RQV labels, which train a controller to score candidate targets without running paired continuations for each one.The task policy is held fixed across each comparison, and the controller predicts from history and target without observing the latent goal.Source: §3.2 · p. 4

3.3 Value-Guided Rollout Allocation

• Value-guided sampling prioritizes a target with high predicted RQV, while a separate coverage draw can test another candidate against a shared PASS reference.The coverage target is sampled from the policy-derived reference distribution over candidates other than the guided target.Source: §3.3 · p. 5

3.4 Policy Learning with Sampling Correction

• PASS-referenced advantages train the policy on individual returns, while arm-inclusion correction counteracts guided sampling and preserves the expected unclipped task-return gradient.The guarantee excludes policy-ratio clipping and reference-policy regularization.Source: §3.4 · pp. 5–6

3.5 Online Co-Training and Inference

• Online updates fit the controller and policy from each shared batch, then refresh the controller under the final policy; at inference, it queries only when predicted RQV is positive.Choosing PASS ends controller intervention, but the task policy may still ask questions on its own.Source: §3.5 · pp. 6–7

4.1 Experimental Setup

• The evaluation spans four benchmarks and ten settings, using Qwen3-4B, official harnesses, configured turn budgets, and means and standard deviations over three seeds.Baselines cover prompting, threshold-based, information-based, supervised, and outcome-based methods.Source: §4.1 · pp. 7–8

4.2 RQ1: Overall Performance

• CLEAT achieves the highest mean score in all ten settings, with gains of 13.24 points on Travel and 6.48 points in IN3 judgment accuracy over the strongest baselines.Gains on held-out Intention and Telepathy support transfer without additional policy or controller training.Source: §4.2 · p. 8

4.3 RQ2: Where the Gain Comes From

• Freezing the controller lowers mean scores by 1.27–6.70 points, while removing value guidance during collection lowers them by 1.97–8.33 points.Both comparisons retain controller-guided collection or controller updates, respectively, isolating the contributions of renewed value supervision and guided rollout allocation.Source: §4.3 · p. 8, Figure 3 · p. 8
• RQV guidance outperforms both unguided collection and information-gain guidance in all six settings; IG guidance lowers Airline by 3.34 points and Telecom by 2.00 points versus unguided collection.The comparison retains the proposal rule and task-return objective; the authors suggest that informative questions may leave fewer turns for execution as one possible explanation for IG’s declines.Source: §4.3 · p. 8, Figure 3 · p. 8
• Removing controller-based selection at inference reduces mean scores by 0.44–6.14 points, yet the trained policy still beats the strongest Travel baseline, 59.49% versus 50.48%.In five of six settings, removing collection guidance causes a larger mean drop than removing controller selection at inference.Source: §4.3 · pp. 8–9

4.4 RQ3: What the Agent Resolves

• CLEAT leads on IN3 instruction recovery at 67.58% and AskBench clarification coverage at 34.87% and 51.49%; these measures include resolving false premises as well as missing details.Judgment accuracy captures whether to ask or proceed, whereas recovery measures how much missing information the exchange resolves; coverage remains below answer accuracy on both AskBench settings.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 RQ4: Performance across Model Scales

• At 1.7B, 4B, and 8B, CLEAT exceeds the backbone and GRPO in AskBench accuracy and coverage; at 8B, accuracy gains over GRPO are 2.08 points on AskMind and 2.50 on AskOverconfidence.Coverage gains at 8B are 4.74 and 5.66 points, respectively; CLEAT’s AskMind accuracy rises with scale, but coverage peaks at 4B.Source: §4.5 · p. 9

• Clarification research spans question answering, code generation, planning, implicit intentions, tool use, incomplete instructions, and communication failures.Value-of-information methods weigh the expected benefit of information against interaction cost.Source: §5 · pp. 9–10
• Interaction-outcome approaches evaluate questions through anticipated or observed dialogue, preference pairs, personalized-response likelihood, and local counterfactual outcomes.Examples include future-turn simulation, CollabLLM, STaR-GATE, ACT, and RICE-PO.Source: §5 · p. 10
• Information-based rewards track belief or decision changes, while experience prioritization allocates training effort; CLEAT combines guided collection with paired task outcomes to train value and policy.Related approaches include IGPO, clarification-specific rewards, InfoPO, and prioritized experience replay.Source: §5 · p. 10

6 Conclusion, Limitations and Future Work

• With Qwen3-4B, CLEAT improves task performance and clarification quality across four benchmarks through co-training the clarification-value controller and task policy.Renewed supervision tracks the task policy’s changing capabilities.Source: §6 · p. 10
• CLEAT’s fixed target constructor and question realizer, repeated-continuation requirement, and simulated users leave real-user effectiveness and evolving goals unestablished.Future work could jointly learn target construction and question realization and reduce the number of paired rollouts needed for value supervision.Source: §6 · p. 10

A.1 Defining Policy-Relative Clarification Value

• Clarification value is policy-relative: the same information can yield different task benefits across policies, and its value can change as the policy improves.Unlike classical value of information for an optimal decision maker, residual query value measures benefit under the policy that actually continues the task.Source: §A.1 · p. 16

A.2 Estimating RQV from Paired Continuations

• Paired QUERY and PASS continuations from the same history, latent goal, and task policy isolate clarification’s effect and provide labels for predicting residual query value.Squared-error regression trains the controller to predict the utility added by clarification rather than the utility attained after asking alone.Source: §A.2 · pp. 16, 18

A.3 Value-Guided Collection under a Fixed Budget

• Under a fixed rollout budget, value-guided selection prioritizes promising targets, a separate coverage draw samples beyond that prioritization, and replication improves comparison precision.Coverage can supply comparisons that revise controller estimates, while replication sharpens the prioritized comparison and retains an additional candidate for evaluation.Source: §A.3 · pp. 18–20

A.4 Policy Learning with Sampling Correction

• PASS-referenced advantages align policy learning with proceeding, while averaging arm contributions and correcting for arm inclusion preserve the task-return objective under guided collection.The query advantage mean is the empirical residual query value; PASS continuations can also improve native task behavior through their execution gradient.Source: §A.4 · pp. 20–22
• Fixed inclusion weights let the controller change which arms receive data without changing their expected task-return gradient weighting.The controller allocates learning opportunities rather than supplying an additional reward; policy advantages use observed task utilities, with correction preserving intended arm weighting.Source: §A.4 · pp. 22–23
• With ϵ = 0.2, every inclusion weight is at most 1.25; coverage broadens supervision while keeping correction weights bounded independently of guidance strength or controller-prediction range.The coverage draw controls correction size independently of how strongly the guided proposal concentrates.Source: §A.4 · p. 22
• Unlike information-based signals that provide intermediate supervision, predicted RQV guides experience collection while the task policy learns from observed utilities.The controller and task policy therefore learn different quantities from the same outcomes.Source: §A.4 · pp. 22–23

A.5 Online Co-Training and Inference

• As the policy improves, better use of an answer can raise its residual value, while better native execution can make the same clarification redundant.This changing comparison motivates renewing controller supervision as the policy evolves.Source: §A.5 · p. 23
• Each shared rollout batch supplies paired RQV labels for fitting the controller and sampling-corrected task-return gradients for updating the policy.The controller guides target selection in the next round, and outcomes from the updated policy provide fresh value supervision.Source: §A.5 · p. 23
• Because controller fitting uses outcomes from π_t but guides collection under π_{t+1}, its supervision has a one-update lag.The next batch renews supervision under π_{t+1}; after the final policy update, fresh paired outcomes refresh the controller without another policy update.Source: §A.5 · p. 23
• At inference, the refreshed controller selects the highest-valued query only when predicted RQV is positive; otherwise it chooses PASS.The task policy retains its own ability to ask even after PASS ends controller intervention for the interaction.Source: §A.5 · p. 23

B Implementation Details

• Training uses the fixed benchmark training pool for up to 120 policy-update steps, with 16 histories per collection step and a 64-task development set for monitoring.The official test split is reserved for final evaluation; controller-only bootstrap and final refresh are additional.Source: B Implementation Details · p. 24
• The controller predicts RQV from encoded history and candidate target features, with trained projection towers and output head over a frozen feature encoder.Its inputs retain the last 1,536 tokens and include the normalized remaining-turn budget.Source: B Implementation Details · p. 24
• At each selected eligible history, fresh paired PASS and QUERY continuations under the current policy provide controller RQV targets and task returns for policy updates.After the contrasted decision, the policy completes each continuation without further controller intervention and may still ask questions itself.Source: B Implementation Details · p. 24
• Task utility follows benchmark-specific evaluator criteria, and shared training settings include benchmark-specific interaction interfaces.Online co-training uses two H100 GPUs, with a Qwen3-30B-A3B model hosted separately for user simulation and evaluation.Source: B Implementation Details · pp. 23–25
• Each batch receives one full-batch policy update using observed task-return advantages and KL regularization toward the frozen backbone.Sampling-corrected contributions are averaged within each sampled arm and then across states; correction weights are held fixed and untruncated.Source: B Implementation Details · p. 25

C.1 Benchmarks and Evaluation Metrics

• UserGym reports Pass-1 scores across Travel, Turtle, Intention, and Telepathy, with Travel and Turtle used for training and Intention and Telepathy held out.Travel uses the highest single-step reward; the other three sum episode rewards, and Intention scores are not percentages and can exceed 100.Source: §C.1 · p. 26
• τ2-Bench reports task pass rates separately for Airline, Retail, and Telecom, where clarification and tool use share an interaction budget.The official evaluator determines whether each task is completed successfully.Source: §C.1 · p. 26
• IN3 reports judgment accuracy, instruction recovery, and vague ask rate to distinguish clarification judgment, resolved missing details, and asking behavior.Judgment accuracy covers all instructions, while recovery and vague ask rate concern underspecified instructions.Source: §C.1 · p. 26
• AskBench reports accuracy and clarification coverage, measuring final correctness separately from resolving every required point before answering.Required points concern missing information in AskMind and false premises in AskOverconfidence.Source: §C.1 · p. 26

C.2 Baselines

• Prompting baselines keep weights fixed, while threshold-based baselines gate clarification using ambiguity, intent uncertainty, or predicted success differences.Thresholds are swept on validation data for each method and benchmark; VoI-Gate thresholds are selected on tuning data and fixed at test time.Source: §C.2 · p. 27
• Information-based baselines use intermediate information signals, whereas CLEAT allocates rollouts by predicted query value and forms policy advantages from task utilities relative to PASS.SFT and GRPO provide supervised and outcome-based references; main comparisons use Qwen3-4B and shared benchmark-specific evaluation protocols.Source: §C.2 · p. 27

D.1 Component Ablations

• Menu augmentation improves native baselines, but the full CLEAT method achieves the highest mean across reported metrics, with particularly large gains over Menu-GRPO on Travel and IN3.The ablations separately assess deployment-time intervention and training-time components; without co-training holds the bootstrapped controller fixed during policy training, while without guidance removes value-guided rollout selection.Source: §D.1 · pp. 27–28

D.2 Comparison with Larger Backbones

• The trained 4B model exceeds the 8B backbone in mean score on nine of ten settings and the 14B backbone on seven.Its IN3 judgment accuracy exceeds the 14B backbone by 13.89 points, while the 14B model remains ahead on Turtle, Retail, and AskOverconfidence.Source: §D.2 · p. 29

D.3 CLEAT across Backbone Sizes

• At 1.7B, CLEAT raises coverage over GRPO from 22.97% to 27.56% on AskMind and from 28.92% to 36.17% on AskOverconfidence.CLEAT has the highest mean on both accuracy and coverage in both AskBench settings at 1.7B, 4B, and 8B.Source: §D.3 · p. 29

D.4 Clarification-Specific Metrics

• CLEAT achieves the highest mean in all three clarification-specific metric columns: IN3 instruction recovery, AskMind coverage, and AskOverconfidence coverage.The strongest baseline varies by metric: SFT leads IN3 recovery, CLAM leads AskMind coverage, and InfoPO leads AskOverconfidence coverage.Source: §D.4 · p. 30, Table 7 · p. 30

D.5 Comparison with ACT and CollabLLM

• CLEAT achieves the highest mean on all six reported metrics, exceeding ACT on AskBench accuracy and coverage and CollabLLM on IN3 recovery and judgment accuracy.Relative to ACT, AskMind and AskOverconfidence accuracy improve by 3.07 and 2.27 points, and coverage by 8.36 and 9.16 points; on IN3, CLEAT exceeds CollabLLM by 11.14 points in instruction recovery and 16.05 points in judgment accuracy.Source: §D.5 · pp. 30–31

D.6 Asking on Underspecified Instructions

• CLEAT reaches an 85.6% vague ask rate and 83.95% overall judgment accuracy, both the highest among displayed methods.Methods with similar asking rates can differ substantially in judgment accuracy, so asking frequency alone does not capture whether agents distinguish underspecified from fully specified instructions.Source: §D.6 · p. 31

D.7 Co-Training Dynamics

• After an initial transient, actor gradient norms settle into a lower range, while KL to the frozen reference rises to about 0.3 on AskOverconfidence and fluctuates around 0.5 later in AskMind training.Empirical RQV moving averages remain positive in both settings as policy, visited histories, and selected targets evolve.Source: §D.7 · pp. 31–32, Figure 8 · p. 31

D.8 Qualitative Examples of Policy Behavior

• In selected trajectories, the trained policy handles a CSV column choice natively and uses indirect airline-preference feedback to select a matching itinerary.The base policy hard-codes the first CSV column in its sampled continuation and produces a malformed airline search, respectively; the examples illustrate that clarification value depends on how the policy handles the task and reply.Source: §D.8 · p. 32

D.9 Cross-Method Clarification Cases

• In the AskOverconfidence case, CLEAT resolves the word-length premise and answers correctly, whereas GRPO and ACT leave it unresolved and answer incorrectly; in AskMind, CLEAT asks for the missing modulus and solves the task.The examples connect clarification success to addressing the source of an error and carrying feedback into the final answer.Source: §D.9 · p. 33

E Additional Related Work

• Related work spans clarification, user assistance, and adaptation methods; CLEAT focuses on acquiring user information according to its incremental value for the evolving task policy.The evaluation uses simulated users, and robustness to real user behavior and evolving goals remains beyond the study’s scope.Source: E Additional Related Work · pp. 33–34

F Prompts

• The prompts separate proposing missing-information targets from selecting QUERY or PASS, with a fixed realizer turning a selected target into a question.A fixed target constructor supplies candidate targets, and the question–answer exchange is appended to the history before task execution resumes.Source: F Prompts · p. 34