Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialogue

• Current VLA-based agents assume instructions are unambiguous, while interactive systems often rely on high-level action primitives and visuomotor policies lack dialogue.These gaps limit reliable collaboration in real-world settings.Source: §I · p. 1
• Ask-to-Clarify detects ambiguity, initiates clarification dialogue, and translates refined intent into motor trajectories.It unifies multi-turn dialogue with low-level visuomotor control without high-level action primitives or external planners.Source: §I · pp. 1–2
• Across 11 real-world tasks, Ask-to-Clarify significantly outperforms existing methods.The authors also report state-of-the-art performance in real-world experiments.Source: §I · p. 2
• Visuomotor fine-tuning can erase dialogue capabilities, motivating a two-stage strategy that decouples cognitive learning from physical manipulation.The authors report that even partial action-stage fine-tuning can completely erase dialogue capabilities.Source: §I · p. 2

A. Embodied Agents for Manipulation

• Hierarchical and parallel architectures pair VLMs with action experts, and Ask-to-Clarify adopts a hierarchical design with an adapter to translate intent into manipulation.Autoregressive VLA action-token generation can be too slow for high-frequency control; specialized action experts are used to enable faster response times.Source: A. Embodied Agents for Manipulation · p. 2

B. Agents that Collaborate with Humans

• Interactive agents can ask users for clarification, but reliance on high-level action primitives limits their suitability for complex, precise manipulation.Ask-to-Clarify targets end-to-end low-level control while retaining dialogue logic through two-stage knowledge-insulation training.Source: B. Agents that Collaborate with Humans · p. 2

III. METHOD

• The method combines a VLM-initialized Cognitive Planner, a diffusion-policy Motor Executor, and an adapter that refines intent using visual features.A two-stage training strategy is used to preserve cognitive capabilities while learning fine-grained visuomotor control.Source: §III · pp. 2–3, Figure 2 · p. 3

• The task begins with an instruction and observation, which may leave ambiguity because of multiple candidate objects or underspecified constraints.The paper gives “Give me the mug” when multiple mugs are present as an example.Source: A. Task Definition · p. 3
• When ambiguity is detected, the agent asks clarification questions and uses user responses over multiple turns to derive a refined instruction.The dialogue history consists of question-answer pairs.Source: A. Task Definition · p. 3
• After resolving intent, the agent switches to execution and generates action chunks to complete the task.Execution conditions on the refined instruction and observation.Source: A. Task Definition · p. 3

• Even partial fine-tuning during action generation can rapidly erase dialogue capabilities, so the strategy separates cognitive learning from visuomotor learning.The paper also identifies conflicting gradients when training cognition and fine-grained motor control together.Source: B. Two-stage Knowledge-insulation… · p. 3
• The first stage uses synthesized ambiguity dialogues to train the planner to detect ambiguity and ask clarification questions.The data pipeline starts with images of objects differing in some but not all attributes, then generates ambiguous instructions, question-answer pairs, and correct instructions.Source: B. Two-stage Knowledge-insulation… · pp. 3–4
• Signal tokens mark ambiguity, resolved instructions, execution readiness, or refusal, and the planner is trained to predict them.In this stage, the vision encoder is frozen and only the planner's LLM parameters are fine-tuned.Source: B. Two-stage Knowledge-insulation… · p. 4
• In the second stage, the adapter combines refined instructions with visual features, while the frozen planner is insulated and the executor learns motor trajectories.The adapter uses affine transformations to produce visually grounded conditioning; the executor trains on expert demonstrations labeled with correct instructions.Source: B. Two-stage Knowledge-insulation… · p. 4

C. Disambiguation Gate

• The training-free Disambiguation Gate routes Cognitive Planner signal tokens to direct system flow.It dynamically switches the agent’s behavior during inference.Source: C. Disambiguation Gate · p. 4
• When the planner detects semantic uncertainty, <AMBG> suspends motor control and prompts a clarification query whose response updates dialogue history.The planner then analyzes the updated context in the next inference cycle.Source: C. Disambiguation Gate · p. 4
• For clear or resolved intent, the planner emits a refined instruction and <NOT AMBG>, then verifies that the target is present and actionable.Source: C. Disambiguation Gate · p. 4
• <ACT> activates the Alignment Adapter and Motor Executor, while <REJ> halts operation for unresolvable conflicts or safety risks such as target absence.Source: C. Disambiguation Gate · p. 4

IV. EXPERIMENTS

• The evaluation first measures collaborative embodied abilities and overall performance against state-of-the-art baselines.Source: §IV · p. 4
• Systematic ablations investigate catastrophic forgetting and test the two-stage knowledge-insulation strategy and Semantic-Visual Alignment Adapter.Source: §IV · p. 4
• Dedicated experiments assess cognitive ability, including through a Dialogue Success Rate metric.Source: §IV · p. 4
• Robustness is tested in out-of-distribution environments with low lighting and distractors.Source: §IV · p. 4

A. Experimental Setup

• The real-world platform uses a 7-DoF xArm 7 with a 1-DoF gripper and wrist-mounted and third-person RealSense D435 cameras.Figure 4 depicts task sequences, randomized object initialization zones, the object inventory, and robot setup.Source: A. Experimental Setup · pp. 4–5, Figure 4 · p. 5
• The real-world suite comprises 11 tasks across Pick-and-Place, Pouring, Stacking, and Insertion, with ambiguous instructions requiring target inference and low-level actions.The task variants include fruit, colored cups, block pairs, and colored cubes.Source: A. Experimental Setup · p. 5
• Task demands progress from basic grasping and repositioning to controlled pouring, precise stacking, and insertion with centimeter-level tolerance.Insertion requires centering the cube above the slot to avoid rim collisions and task failure.Source: A. Experimental Setup · p. 5
• Stage 1 uses 1,150 synthesized, manually verified multi-turn dialogues from 410 initial trajectory frames; stage 2 uses 410 expert demonstrations across 11 tasks.Each dialogue has an ambiguous instruction, 1–3 clarification QA rounds, and a resolved instruction; demonstrations are recorded at 30 Hz.Source: A. Experimental Setup · p. 5
• Qwen2-VL-2B-Instruct instantiates the Cognitive Planner and ScaleDP-Huge the Motor Executor, connected by an adapter using learned affine modulation.Two MLPs map observation embeddings to channel-wise scale and shift parameters applied to prompt embeddings; the action chunk size is 50 timesteps.Source: A. Experimental Setup · p. 5

B. Baselines

• The five baselines cover standard VLAs, a primitive-based ASK-TO-ACT approach, and a decoupled large-language-planner plus VLA-policy pipeline.Source: B. Baselines · pp. 5–6
• π0 uses a parallel flow-matching action expert, π0-FAST tokenizes continuous actions autoregressively, and OpenVLA-OFT uses a hierarchical diffusion action expert.Source: B. Baselines · p. 5
• LLM-Primitives resolves ambiguity through dialogue, then uses object detection, segmentation, depth-based 3D poses, and open-loop inverse-kinematics trajectories.Source: B. Baselines · p. 6
• LLM-Policy uses a large planner to reformulate dialogue into a precise text instruction for a separately optimized, fine-tuned π0 policy.Source: B. Baselines · p. 6
• The three standard VLAs receive unambiguous instructions, while LLM-Primitives and LLM-Policy face the same ambiguous instructions as Ask-to-Clarify.Source: B. Baselines · p. 6
• Task Success Rate measures completion of both correct target identification and manipulation, while Disambiguation Success Rate isolates correct ambiguity resolution and counts correct absent-target refusals.Source: B. Baselines · p. 6

C. Real-world Evaluation

• 8.6% success: LLM-Primitives falls to 0.0% on Insertion, while Ask-to-Clarify reaches 50.0% using closed-loop visuomotor control.The paper attributes primitive-based errors to pose-estimation inaccuracies propagating through deterministic trajectory planning.Source: C. Real-world Evaluation · p. 6
• Ask-to-Clarify achieves 82.7% success across 11 real-world tasks while resolving vague commands through multi-turn dialogue, matching LLM-Policy’s 78.2% success.LLM-Policy couples a 72B-parameter planner with fine-tuned π0 for execution; Ask-to-Clarify uses a 2B Cognitive Planner.Source: C. Real-world Evaluation · p. 6, Figure 5 · p. 7
• 93% Pick-and-Place success required freezing the planner and optimizing the adapter and executor; omitting the adapter or fine-tuning the planner led to failure.A mean-pooling bridge also yielded 0% success; planner fine-tuning erased Stage 1 dialogue capabilities.Source: C. Real-world Evaluation · p. 7, Figure 6 · p. 7
• The 2B planner matches the structured 72B model across Present and Absent settings, with the lowest inference latency and correct refusals for missing targets.The comparison reports that the 2B planner has the highest DSR across the two settings and matches the baseline Qwen2-VL-2B latency.Source: C. Real-world Evaluation · pp. 7–8, Figure 7 · p. 7
• Under 50% reduced illumination, Ask-to-Clarify’s Stacking success falls from 90.0% to 80.0%, versus LLM-Policy’s drop from 92.5% to 22.5%.The authors suggest the frozen VLM vision encoder may preserve robustness to visual distribution shifts.Source: Figure 8 · p. 8, C. Real-world Evaluation · p. 8
• With similar-looking distractors present, Ask-to-Clarify achieves 80.0% success, compared with 65.0% for LLM-Policy.The test places an apple and a pomegranate on the table.Source: C. Real-world Evaluation · p. 8, Figure 8 · p. 8

V. CONCLUSION

• Ask-to-Clarify integrates a VLM-based Cognitive Planner and Diffusion-based Motor Executor, linked by a Semantic-Visual Alignment Adapter and coordinated at inference by a Disambiguation Gate.The two-stage strategy freezes the planner during visuomotor learning to preserve previously acquired knowledge.Source: §V · p. 8
• Future work will add long-term memory for user preferences and proactive intent anticipation, and explore natural voice communication.Source: §V · p. 8