Source-linked AI summary
Active Perception for Embodied Disambiguation
Yiwei Liu, Luwei Yang
TL;DR
Robots may lack task-relevant physical evidence, not just user intent, when disambiguating targets in embodied environments. This paper uses active observation with vision-language reasoning to acquire evidence, seek clarification, or select a target, and real-robot experiments show unified physical and interactive disambiguation.
Problem
Embodied target disambiguation must address both incomplete observations and unresolved user intent, since clarification cannot reveal targets or attributes absent from the robot’s view.
Method
A vision-language framework uses active observation as its information backbone, choosing whether to acquire another view, query the user, or commit to target selection.
Results
Real-robot experiments show active perception recovers missing visual evidence, expands useful observation, and improves physical grounding for clarification when needed.
Takeaways & Limitations
Embodied disambiguation can combine physical information acquisition with user-intent clarification within one process.
Takeaways & Limitations
The framework does not address substantial conflicts between physical observations and user feedback, limiting its conclusions to processes without such inconsistency.
Abstract
from arXiv · showhide
Natural language provides robots with a flexible task interface, but target ambiguity in embodied environments arises not only from user intent; it can also result from missing taskrelevant physical evidence in the current observation. Existing interactive disambiguation methods primarily obtain additional information by asking the user, whereas occlusion, restricted viewpoints, unreadable text, and unobserved targets require the robot to actively change its observation. We propose an active-perception framework for embodied target disambiguation that uses active observation as the backbone for information acquisition and uses a vision-language model to decide, on the basis of accumulated visual evidence and interaction information, whether to continue observing, request clarification, or complete target selection. Active observation can both directly recover missing discriminative evidence and reveal object names, labels, and semantic attributes, thereby improving user clarification when it remains necessary. Real-robot experiments show that the framework combines physical information acquisition and userintent clarification within a unified embodied disambiguation process.
I. Introduction
The paper frames embodied target disambiguation as requiring both active acquisition of missing physical evidence and clarification of unresolved user intent. It proposes a vision-language framework that uses active observation as the information backbone and validates it in real-robot experiments.
- Problem formulation: Embodied target ambiguity can arise from incomplete user descriptions, multiple matching objects, or insufficient visual input, motivating clarification and active observation.Missing physical evidence may result from occlusion, unreadable text, restricted viewpoints, or targets outside the useful observation range.
- Problem formulation: The paper distinguishes observation incompleteness, where physical evidence is unobserved, from semantic or intent ambiguity, where user preference remains unknown.The former requires changing the observation, while the latter requires clarification.
- Framework: The proposed vision-language framework continuously acquires visual evidence and reasons over instructions, accumulated observations, and clarification information before continuing observation, asking questions, or selecting a target.Active observation first updates the robot’s physical-scene understanding and then improves the semantic basis for clarification.
- Experiments: Real-robot experiments evaluate active perception for recovering missing evidence, improving clarification with acquired semantic attributes, and discovering targets outside the initial useful view.These experiments cover new viewpoints, semantic attributes, and changes to the observed region.
II. Related Work · A. Interactive Disambiguation for Robotic Manipulation
Interactive robotic disambiguation evolved from question-centered visual grounding toward manipulation-integrated clarification, attribute-based questioning, uncertainty-aware interaction, flexible communication, and vision–language reasoning. The present study builds on fixed-view SeeAsk asking strategies while addressing ambiguity through embodied information acquisition.
- A. Interactive Disambiguation for Robotic Manipulation: Early interactive visual grounding showed that robots can ask for additional information when multiple image regions remain compatible with a referring expression.INGRESS systematized this process through candidate generation, relational grounding, and iterative object-specific clarification.
- A. Interactive Disambiguation for Robotic Manipulation: INVIGORATE integrated visual grounding, blocking relationships, question generation, and grasping, embedding ambiguity resolution within a physical manipulation task.Attribute-guided disambiguation further treated question selection as a decision variable by choosing attributes that separate remaining candidates.
- A. Interactive Disambiguation for Robotic Manipulation: SeeAsk extended interactive disambiguation to open-set objects and open-vocabulary interaction in real robotic grasping.Its greedy baseline confirms one preferred candidate at a time, whereas its static baseline asks using attributes available before active camera motion.
- A. Interactive Disambiguation for Robotic Manipulation: KnowNo and CLARA shifted attention from what to ask toward when clarification is warranted for reliable execution.KnowNo addressed uncertainty alignment, while CLARA distinguished clear, ambiguous, and infeasible commands before user interaction.
- A. Interactive Disambiguation for Robotic Manipulation: PROGrasp broadened the interaction channel by interpreting intention-oriented and freer-form responses instead of relying on fixed yes/no or option-ID protocols.This work made human–robot communication less rigid than standardized answer formats.
- A. Interactive Disambiguation for Robotic Manipulation: AmbResVLM incorporated vision–language reasoning into robotic ambiguity resolution through detection, clarification generation, answer interpretation, and robotic validation.It represents a recent pipeline combining language-model reasoning with embodied validation.
B. From Asking for Information to Acquiring It from the World
Target ambiguity can arise from missing physical evidence as well as unstated user preferences or intentions. The framework therefore treats embodied visual information acquisition as complementary to linguistic clarification, asking whether uncertainty should first be resolved from the world or the user.
- Information sources: Human responses are the common information channel when ambiguity concerns a user’s unstated preference or intention.The robot observes a scene, forms a candidate hypothesis, and requests a linguistic distinction from the user.
- Information sources: Occlusion, unreadable text, reversed surfaces, and targets outside useful local observation can create ambiguity that language reasoning cannot resolve.User answers may reduce uncertainty among observed alternatives but cannot reveal an alternative absent from the robot’s observation.
- Information sources: The robot can change its viewpoint, obtain a fresh RGB-D observation, and then determine whether remaining uncertainty concerns the world or user intent.Embodied observation supplies information unavailable from a fixed image.
- Information sources: The central decision is whether missing information should first be acquired from the physical world or requested from the user.VLM-based ambiguity resolution provides the language-and-visual reasoning substrate for this interaction.
III. Problem Formulation · A. Embodied Target Disambiguation · B. Two Sources of Embodied Ambiguity
The robot resolves an intended physical target by combining visual evidence with embodied interaction, choosing whether to observe, ask the user, or commit. Embodied ambiguity arises from incomplete physical evidence or unspecified user preferences.
- A. Embodied Target Disambiguation: The VLM selects α ∈ {O, Q, S}: acquire another camera observation, query the user for clarification, or commit to the resolved physical target.These actions combine visual evidence with information obtained through embodied interaction.
- B. Two Sources of Embodied Ambiguity: Observation incompleteness occurs when the current view lacks task-relevant physical evidence needed to resolve the target.Examples include occlusion, insufficient viewpoints, unreadable printed text, or an unexposed relevant side.
- B. Two Sources of Embodied Ambiguity: Incomplete observations may fail to distinguish candidates already supported by C(o) or omit a relevant physical feature.The passage defines C(o) as the candidate hypothesis supported by observation o.
- B. Two Sources of Embodied Ambiguity: Semantic or intent ambiguity occurs when the environment is sufficiently visible but the instruction does not specify the desired functional or semantic property.Active observation can improve evidence for asking, but cannot replace the missing preference.
IV. Active Perception for Embodied Disambiguation … C. Observation-Grounded Target Selection
The framework acquires and accumulates embodied visual evidence, using a vision-language model to decide whether to observe again, ask clarification, or commit to a target. Active observation can reveal task-relevant physical and semantic evidence, grounding clarification before consistent target selection.
- IV. Active Perception for Embodied Disambiguation: The process centers on acquiring and accumulating embodied visual evidence for target disambiguation.The VLM reasons from the user instruction, current camera view, prior observations, and clarification dialogue.
- C. Observation-Grounded Target Selection: After each fresh view, the VLM may continue observing, formulate evidence-grounded clarification, or commit to a resolved target.These alternatives unify physical information acquisition, user clarification, and terminal target selection.
- A. Active Observation as the Information Backbone: When task-relevant information is insufficient, the VLM chooses α = O to obtain a fresh RGB-D observation from a changed camera viewpoint.The eye-in-hand camera changes the physical evidence available to subsequent reasoning.
- A. Active Observation as the Information Backbone: The VLM can repeat active observation when newly acquired evidence remains incomplete.After each fresh view, it reassesses the scene and may choose another observation.
- B. Observation-Grounded Clarification: Additional observations can expose medicine names, printed package labels, object identities, and other meaningful semantic attributes.An initial view may provide only a coarse description, while a later view can reveal discriminative semantic evidence.
- B. Observation-Grounded Clarification: Clarification is generated from the visual understanding built through observation rather than from the initial view alone.Active observation changes the semantic evidence from which clarification is formed.
- C. Observation-Grounded Target Selection: Once accumulated visual evidence and any required clarification identify a target consistently, the VLM chooses α = S and commits to that physical target.The information flow may include repeated observations, or observation followed by grounded clarification and then selection.
V. Real-Robot Experiments · A. Platform and Protocol
The real-robot experiments use a PiPER manipulator with an eye-in-hand DABAI DC1 RGB-D camera on tabletop scenes. Matched trials control scene, observation, and instruction conditions while evaluating ambiguity resolution before target commitment.
- A. Platform and Protocol: The platform is a PiPER six-degree-of-freedom manipulator equipped with a DABAI DC1 RGB-D camera in an eye-in-hand configuration.The setup is depicted in Fig. 2.
- A. Platform and Protocol: The robot operates on real tabletop scenes and acquires synchronized RGB-D observations before and after active camera motion.This protocol supports active observation during the experiments.
- A. Platform and Protocol: Each matched comparison uses the same physical scene configuration and the same initial observation condition.These controls keep the compared methods aligned at the start of each trial.
- A. Platform and Protocol: All matched trials use the same user instruction across methods.The comparison therefore holds the instruction constant across matched methods.
- A. Platform and Protocol: All matched trials use Qwen3.7-Plus through an Alibaba Cloud Model Studio workspace in the China (Beijing, cn-beijing) region.The model ID is qwen3.7-plus.
- A. Platform and Protocol: The protocol evaluates ambiguity resolution before target commitment.Active observation provides visual evidence for subsequent clarification and target selection.
B. Baselines · C. Matched Scenarios and Research Questions
The evaluation compares passive selection and two nonmoving asking baselines across nine matched real-robot scenarios. It investigates whether active perception recovers visual evidence, improves clarification grounding, and expands the observed workspace.
- B. Baselines: Passive Direct Selection uses only the initial observation, asks no question, and directly selects one candidate.
- B. Baselines: Greedy Asking does not move the camera and sequentially asks confirmation-style questions about currently preferred candidates.
- B. Baselines: Static Asking does not move the camera and constructs candidate-partition questions from attributes visible in the frozen initial view.
- C. Matched Scenarios and Research Questions: The comparison includes nine matched scenarios and four methods per scenario, totaling 36 real-robot trials.Table I summarizes scene configuration, ambiguity regime, requested evidence, and outcomes.
- C. Matched Scenarios and Research Questions: RQ1 tests whether active perception recovers discriminative visual evidence for a target already represented in the initial local observation.
- C. Matched Scenarios and Research Questions: RQ2 tests whether active perception provides better semantic grounding for a clarification that remains necessary.
- C. Matched Scenarios and Research Questions: RQ3 tests whether active perception expands the observed workspace when the requested target is absent from the initial candidate set.
D. Evaluation · E. Recovering Discriminative Evidence for Initially Observed Targets · F. Semantic Grounding for User Clarification
The evaluation measures target resolution, grounding source, clarification interaction, and active perception across physical-evidence and semantic-intent scenarios. Results show that active observation recovers missing discriminative evidence and improves clarification by exposing semantic information, while passive selection can remain unsupported or incorrect.
- D. Evaluation: Evaluation tracks target resolution, grounding source, clarification questions and exchanges, and the number of new viewpoints obtained before selection.Grounding distinguishes fresh visual evidence, user clarification, initial-view evidence, and unsupported selection.
- E. Recovering Discriminative Evidence for Initially Observed Targets: In A1–A3, Proposed acquired occluded, finetext, and volume-specific attributes from fresh views, while Passive failed in the latter two cases.Greedy and Static resolved these targets through user clarification without acquiring new visual observations.
- E. Recovering Discriminative Evidence for Initially Observed Targets: All five physical scenarios contained the requested target initially, but their first observations lacked sufficient discriminative evidence for reliable resolution.The results therefore evaluate recovery of evidence that was physically present but initially unavailable for dependable selection.
- F. Semantic Grounding for User Clarification: Active observation changes which semantic attributes are available for clarification while the physical scene remains visible and the desired medicine is underspecified.Table II consolidates the recorded questions and user responses with original Chinese records and faithful English translations.
- F. Semantic Grounding for User Clarification: In A6, active surveying exposes package names and semantic attributes, enabling Proposed to ask with medicine names while Greedy and Static use less semantic descriptions.The user response resolves each interactive trial; Passive asks no question and remains an unsupported direct selection.
- F. Semantic Grounding for User Clarification: In A7, free-form naming resolves the target for Proposed, Greedy, and Static, whereas Passive remains an unsupported incorrect guess.Greedy repeats appearance-based confirmation after the user names the medicine, failing to exploit the richer semantic information in the answer.
G. Searching Beyond the Initial Observation · VI. Discussion · A. Not Every Ambiguity Should Trigger a Question
The experiments show that active observation can expand target search beyond the initial view, while clarification addresses ambiguity that remains when the scene is visible but user preference is unspecified. The useful next action depends on whether uncertainty comes from missing world evidence or missing intent.
- G. Searching Beyond the Initial Observation: The Proposed method recognized the requested medicine was absent locally, continued searching after user reiteration, and selected it from a later fresh view.Fixed-view methods remained confined to the observed alternatives.
- G. Searching Beyond the Initial Observation: A8 and A9 show that changing the observation can expand candidate hypotheses beyond the initially visible local set.Both cases required searching beyond the initial observation to identify the requested target.
- G. Searching Beyond the Initial Observation: For A9, the Proposed method obtained a fresh view, read the requested label, and selected the distant target.Passive, Greedy, and Static made zero pre-selection active moves and did not resolve it.
- VI. Discussion: The experiments identify active observation and clarification as complementary responses to ambiguity.The source of uncertainty determines which next action is useful.
- A. Not Every Ambiguity Should Trigger a Question: When missing information lies in the world, active observation supplies evidence from occluded surfaces, unreadable text, or unobserved regions.These cases concern physical evidence unavailable in the current scene view.
- A. Not Every Ambiguity Should Trigger a Question: When the scene is sufficiently visible but user preference remains unspecified, clarification supplies the missing intent.Not every ambiguity therefore requires an observation change.
B. Seeing and Asking Provide Different Information · C. Active Perception Can Improve the Language Interface · D. Scope of Pretrained Semantic Knowledge
Active observation and user clarification provide complementary information for embodied target disambiguation: seeing reveals physical evidence, while asking reveals intent. The framework uses active perception to ground language interaction, while recognizing that current-interaction evidence exceeds pretrained semantic knowledge.
- B. Seeing and Asking Provide Different Information: Active observation identifies what is physically present, while user clarification specifies what the user actually wants.The two information channels are complementary because visual evidence identifies available targets and user responses identify the intended one.
- C. Active Perception Can Improve the Language Interface: Fresh views expose package names, colors, stripes, graphics, and other attributes that support more grounded clarification questions.These observations replace fixed-view positional proxies with task-relevant visual descriptions.
- C. Active Perception Can Improve the Language Interface: Active perception can improve the language interface without eliminating the need for a question.The A6 and A7 traces show that active observation and user interaction can remain sequentially complementary.
- C. Active Perception Can Improve the Language Interface: Grounded questions let users contribute preferences or intentions, including through free-form answers.A semantically informative user response can still be wasted by a rigid interaction process.
- D. Scope of Pretrained Semantic Knowledge: The evaluated tasks require current-interaction evidence that pretrained VLM knowledge alone cannot resolve.Examples include a letter attached to a box, a volume marker on one of two similar books, and whether a medicine box is present in the workspace.
- D. Scope of Pretrained Semantic Knowledge: A VLM may understand object categories and word meanings while lacking decisive evidence specific to the currently observed scene.The distinction motivates combining visual evidence, user feedback, and prior knowledge during sequential embodied interaction.
E. Consistency Between Physical Evidence and User Feedback · VII. Conclusion
The framework assigns complementary roles to active observation and user clarification in embodied target disambiguation, while acknowledging that conflicts between physical evidence and user feedback remain unaddressed. The robot uses updated visual understanding to continue observing, request clarification, or complete target selection.
- E. Consistency Between Physical Evidence and User Feedback: Active observation provides physical evidence about the current environment, while user clarification supplies intent or preference information unavailable from visual evidence.
- E. Consistency Between Physical Evidence and User Feedback: The paper primarily considers cases where physical observations and user feedback are mutually consistent.
- E. Consistency Between Physical Evidence and User Feedback: Conflicting evidence sources may reflect different observations and prior knowledge rather than an error by the user or visual system.
- E. Consistency Between Physical Evidence and User Feedback: Resolving conflicts would require assessing source reliability and choosing among renewed observation, further clarification, verification, or deferred target selection.
- E. Consistency Between Physical Evidence and User Feedback: The paper does not address conflicts between physical evidence and user feedback.
- VII. Conclusion: The proposed vision-language framework uses active observation as the information backbone for embodied target disambiguation.
- VII. Conclusion: The robot changes viewpoint to acquire physical evidence missing from its current observation.
- VII. Conclusion: Based on updated visual understanding, the robot continues observing, requests user clarification, or completes target selection.