PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Perception for Embodied Disambiguation

Yiwei Liu, Luwei Yang

arXiv:2608.13605v1cs.AIcs.RO

TL;DR

Robots may lack task-relevant physical evidence, not just user intent, when disambiguating targets in embodied environments. This paper uses active observation with vision-language reasoning to acquire evidence, seek clarification, or select a target, and real-robot experiments show unified physical and interactive disambiguation.

Abstract

from arXiv · show

Natural language provides robots with a flexible task interface, but target ambiguity in embodied environments arises not only from user intent; it can also result from missing taskrelevant physical evidence in the current observation. Existing interactive disambiguation methods primarily obtain additional information by asking the user, whereas occlusion, restricted viewpoints, unreadable text, and unobserved targets require the robot to actively change its observation. We propose an active-perception framework for embodied target disambiguation that uses active observation as the backbone for information acquisition and uses a vision-language model to decide, on the basis of accumulated visual evidence and interaction information, whether to continue observing, request clarification, or complete target selection. Active observation can both directly recover missing discriminative evidence and reveal object names, labels, and semantic attributes, thereby improving user clarification when it remains necessary. Real-robot experiments show that the framework combines physical information acquisition and userintent clarification within a unified embodied disambiguation process.

I. Introduction

The paper frames embodied target disambiguation as requiring both active acquisition of missing physical evidence and clarification of unresolved user intent. It proposes a vision-language framework that uses active observation as the information backbone and validates it in real-robot experiments.

Problem formulation

  • Embodied target ambiguity can arise from incomplete user descriptions, multiple matching objects, or insufficient visual input, motivating clarification and active observation.Missing physical evidence may result from occlusion, unreadable text, restricted viewpoints, or targets outside the useful observation range.Source: §I · p. 1
  • The paper distinguishes observation incompleteness, where physical evidence is unobserved, from semantic or intent ambiguity, where user preference remains unknown.The former requires changing the observation, while the latter requires clarification.Source: §I · pp. 1–2

Framework

  • The proposed vision-language framework continuously acquires visual evidence and reasons over instructions, accumulated observations, and clarification information before continuing observation, asking questions, or selecting a target.Active observation first updates the robot’s physical-scene understanding and then improves the semantic basis for clarification.Source: §I · pp. 1–2

Experiments

  • Real-robot experiments evaluate active perception for recovering missing evidence, improving clarification with acquired semantic attributes, and discovering targets outside the initial useful view.These experiments cover new viewpoints, semantic attributes, and changes to the observed region.Source: §I · p. 2

II. Related Work · A. Interactive Disambiguation for Robotic Manipulation

Interactive robotic disambiguation evolved from question-centered visual grounding toward manipulation-integrated clarification, attribute-based questioning, uncertainty-aware interaction, flexible communication, and vision–language reasoning. The present study builds on fixed-view SeeAsk asking strategies while addressing ambiguity through embodied information acquisition.

A. Interactive Disambiguation for Robotic Manipulation

  • Early interactive visual grounding showed that robots can ask for additional information when multiple image regions remain compatible with a referring expression.INGRESS systematized this process through candidate generation, relational grounding, and iterative object-specific clarification.Source: A. Interactive Disambiguation for… · p. 2
  • INVIGORATE integrated visual grounding, blocking relationships, question generation, and grasping, embedding ambiguity resolution within a physical manipulation task.Attribute-guided disambiguation further treated question selection as a decision variable by choosing attributes that separate remaining candidates.Source: A. Interactive Disambiguation for… · p. 2
  • SeeAsk extended interactive disambiguation to open-set objects and open-vocabulary interaction in real robotic grasping.Its greedy baseline confirms one preferred candidate at a time, whereas its static baseline asks using attributes available before active camera motion.Source: A. Interactive Disambiguation for… · p. 2
  • KnowNo and CLARA shifted attention from what to ask toward when clarification is warranted for reliable execution.KnowNo addressed uncertainty alignment, while CLARA distinguished clear, ambiguous, and infeasible commands before user interaction.Source: A. Interactive Disambiguation for… · p. 2
  • PROGrasp broadened the interaction channel by interpreting intention-oriented and freer-form responses instead of relying on fixed yes/no or option-ID protocols.This work made human–robot communication less rigid than standardized answer formats.Source: A. Interactive Disambiguation for… · p. 2
  • AmbResVLM incorporated vision–language reasoning into robotic ambiguity resolution through detection, clarification generation, answer interpretation, and robotic validation.It represents a recent pipeline combining language-model reasoning with embodied validation.Source: A. Interactive Disambiguation for… · p. 2

B. From Asking for Information to Acquiring It from the World

Target ambiguity can arise from missing physical evidence as well as unstated user preferences or intentions. The framework therefore treats embodied visual information acquisition as complementary to linguistic clarification, asking whether uncertainty should first be resolved from the world or the user.

Information sources

  • Human responses are the common information channel when ambiguity concerns a user’s unstated preference or intention.The robot observes a scene, forms a candidate hypothesis, and requests a linguistic distinction from the user.Source: B. From Asking for Information to… · p. 2
  • Occlusion, unreadable text, reversed surfaces, and targets outside useful local observation can create ambiguity that language reasoning cannot resolve.User answers may reduce uncertainty among observed alternatives but cannot reveal an alternative absent from the robot’s observation.Source: B. From Asking for Information to… · p. 2
  • The robot can change its viewpoint, obtain a fresh RGB-D observation, and then determine whether remaining uncertainty concerns the world or user intent.Embodied observation supplies information unavailable from a fixed image.Source: B. From Asking for Information to… · pp. 2–3
  • The central decision is whether missing information should first be acquired from the physical world or requested from the user.VLM-based ambiguity resolution provides the language-and-visual reasoning substrate for this interaction.Source: B. From Asking for Information to… · pp. 2–3

III. Problem Formulation · A. Embodied Target Disambiguation · B. Two Sources of Embodied Ambiguity

The robot resolves an intended physical target by combining visual evidence with embodied interaction, choosing whether to observe, ask the user, or commit. Embodied ambiguity arises from incomplete physical evidence or unspecified user preferences.

A. Embodied Target Disambiguation

  • The VLM selects α ∈ {O, Q, S}: acquire another camera observation, query the user for clarification, or commit to the resolved physical target.These actions combine visual evidence with information obtained through embodied interaction.Source: A. Embodied Target Disambiguation · p. 3

B. Two Sources of Embodied Ambiguity

  • Observation incompleteness occurs when the current view lacks task-relevant physical evidence needed to resolve the target.Examples include occlusion, insufficient viewpoints, unreadable printed text, or an unexposed relevant side.Source: B. Two Sources of Embodied Ambiguity · p. 3
  • Incomplete observations may fail to distinguish candidates already supported by C(o) or omit a relevant physical feature.The passage defines C(o) as the candidate hypothesis supported by observation o.Source: B. Two Sources of Embodied Ambiguity · p. 3
  • Semantic or intent ambiguity occurs when the environment is sufficiently visible but the instruction does not specify the desired functional or semantic property.Active observation can improve evidence for asking, but cannot replace the missing preference.Source: B. Two Sources of Embodied Ambiguity · p. 3

IV. Active Perception for Embodied Disambiguation … C. Observation-Grounded Target Selection

The framework acquires and accumulates embodied visual evidence, using a vision-language model to decide whether to observe again, ask clarification, or commit to a target. Active observation can reveal task-relevant physical and semantic evidence, grounding clarification before consistent target selection.

IV. Active Perception for Embodied Disambiguation

  • The process centers on acquiring and accumulating embodied visual evidence for target disambiguation.The VLM reasons from the user instruction, current camera view, prior observations, and clarification dialogue.Source: §IV · p. 3

C. Observation-Grounded Target Selection

  • After each fresh view, the VLM may continue observing, formulate evidence-grounded clarification, or commit to a resolved target.These alternatives unify physical information acquisition, user clarification, and terminal target selection.Source: §IV · p. 3, C. Observation-Grounded Target Selection · p. 3

A. Active Observation as the Information Backbone

  • When task-relevant information is insufficient, the VLM chooses α = O to obtain a fresh RGB-D observation from a changed camera viewpoint.The eye-in-hand camera changes the physical evidence available to subsequent reasoning.Source: A. Active Observation as the… · p. 3
  • The VLM can repeat active observation when newly acquired evidence remains incomplete.After each fresh view, it reassesses the scene and may choose another observation.Source: A. Active Observation as the… · p. 3, §IV · p. 3

B. Observation-Grounded Clarification

  • Additional observations can expose medicine names, printed package labels, object identities, and other meaningful semantic attributes.An initial view may provide only a coarse description, while a later view can reveal discriminative semantic evidence.Source: B. Observation-Grounded Clarification · p. 3
  • Clarification is generated from the visual understanding built through observation rather than from the initial view alone.Active observation changes the semantic evidence from which clarification is formed.Source: B. Observation-Grounded Clarification · p. 3

C. Observation-Grounded Target Selection

  • Once accumulated visual evidence and any required clarification identify a target consistently, the VLM chooses α = S and commits to that physical target.The information flow may include repeated observations, or observation followed by grounded clarification and then selection.Source: C. Observation-Grounded Target Selection · p. 3

V. Real-Robot Experiments · A. Platform and Protocol

The real-robot experiments use a PiPER manipulator with an eye-in-hand DABAI DC1 RGB-D camera on tabletop scenes. Matched trials control scene, observation, and instruction conditions while evaluating ambiguity resolution before target commitment.

A. Platform and Protocol

  • The platform is a PiPER six-degree-of-freedom manipulator equipped with a DABAI DC1 RGB-D camera in an eye-in-hand configuration.The setup is depicted in Fig. 2.Source: A. Platform and Protocol · p. 3, Figure 2 · p. 4
  • The robot operates on real tabletop scenes and acquires synchronized RGB-D observations before and after active camera motion.This protocol supports active observation during the experiments.Source: A. Platform and Protocol · p. 3
  • Each matched comparison uses the same physical scene configuration and the same initial observation condition.These controls keep the compared methods aligned at the start of each trial.Source: A. Platform and Protocol · p. 3
  • All matched trials use the same user instruction across methods.The comparison therefore holds the instruction constant across matched methods.Source: A. Platform and Protocol · p. 4
  • All matched trials use Qwen3.7-Plus through an Alibaba Cloud Model Studio workspace in the China (Beijing, cn-beijing) region.The model ID is qwen3.7-plus.Source: A. Platform and Protocol · p. 4
  • The protocol evaluates ambiguity resolution before target commitment.Active observation provides visual evidence for subsequent clarification and target selection.Source: A. Platform and Protocol · p. 4, Figure 1 · p. 4

B. Baselines · C. Matched Scenarios and Research Questions

The evaluation compares passive selection and two nonmoving asking baselines across nine matched real-robot scenarios. It investigates whether active perception recovers visual evidence, improves clarification grounding, and expands the observed workspace.

B. Baselines

  • Passive Direct Selection uses only the initial observation, asks no question, and directly selects one candidate.Source: B. Baselines · p. 4
  • Greedy Asking does not move the camera and sequentially asks confirmation-style questions about currently preferred candidates.Source: B. Baselines · p. 4
  • Static Asking does not move the camera and constructs candidate-partition questions from attributes visible in the frozen initial view.Source: B. Baselines · p. 4

C. Matched Scenarios and Research Questions

D. Evaluation · E. Recovering Discriminative Evidence for Initially Observed Targets · F. Semantic Grounding for User Clarification

The evaluation measures target resolution, grounding source, clarification interaction, and active perception across physical-evidence and semantic-intent scenarios. Results show that active observation recovers missing discriminative evidence and improves clarification by exposing semantic information, while passive selection can remain unsupported or incorrect.

D. Evaluation

  • Evaluation tracks target resolution, grounding source, clarification questions and exchanges, and the number of new viewpoints obtained before selection.Grounding distinguishes fresh visual evidence, user clarification, initial-view evidence, and unsupported selection.Source: D. Evaluation · p. 4

E. Recovering Discriminative Evidence for Initially Observed Targets

  • In A1–A3, Proposed acquired occluded, finetext, and volume-specific attributes from fresh views, while Passive failed in the latter two cases.Greedy and Static resolved these targets through user clarification without acquiring new visual observations.Source: E. Recovering Discriminative Evidence… · p. 4
  • All five physical scenarios contained the requested target initially, but their first observations lacked sufficient discriminative evidence for reliable resolution.The results therefore evaluate recovery of evidence that was physically present but initially unavailable for dependable selection.Source: E. Recovering Discriminative Evidence… · p. 4

F. Semantic Grounding for User Clarification

  • Active observation changes which semantic attributes are available for clarification while the physical scene remains visible and the desired medicine is underspecified.Table II consolidates the recorded questions and user responses with original Chinese records and faithful English translations.Source: F. Semantic Grounding for User… · p. 5
  • In A6, active surveying exposes package names and semantic attributes, enabling Proposed to ask with medicine names while Greedy and Static use less semantic descriptions.The user response resolves each interactive trial; Passive asks no question and remains an unsupported direct selection.Source: F. Semantic Grounding for User… · p. 5
  • In A7, free-form naming resolves the target for Proposed, Greedy, and Static, whereas Passive remains an unsupported incorrect guess.Greedy repeats appearance-based confirmation after the user names the medicine, failing to exploit the richer semantic information in the answer.Source: F. Semantic Grounding for User… · p. 5

G. Searching Beyond the Initial Observation · VI. Discussion · A. Not Every Ambiguity Should Trigger a Question

The experiments show that active observation can expand target search beyond the initial view, while clarification addresses ambiguity that remains when the scene is visible but user preference is unspecified. The useful next action depends on whether uncertainty comes from missing world evidence or missing intent.

G. Searching Beyond the Initial Observation

  • The Proposed method recognized the requested medicine was absent locally, continued searching after user reiteration, and selected it from a later fresh view.Fixed-view methods remained confined to the observed alternatives.Source: G. Searching Beyond the Initial… · p. 5
  • A8 and A9 show that changing the observation can expand candidate hypotheses beyond the initially visible local set.Both cases required searching beyond the initial observation to identify the requested target.Source: G. Searching Beyond the Initial… · p. 5
  • For A9, the Proposed method obtained a fresh view, read the requested label, and selected the distant target.Passive, Greedy, and Static made zero pre-selection active moves and did not resolve it.Source: G. Searching Beyond the Initial… · p. 5

VI. Discussion

A. Not Every Ambiguity Should Trigger a Question

  • When missing information lies in the world, active observation supplies evidence from occluded surfaces, unreadable text, or unobserved regions.These cases concern physical evidence unavailable in the current scene view.Source: A. Not Every Ambiguity Should Trigger… · p. 5
  • When the scene is sufficiently visible but user preference remains unspecified, clarification supplies the missing intent.Not every ambiguity therefore requires an observation change.Source: A. Not Every Ambiguity Should Trigger… · p. 5

B. Seeing and Asking Provide Different Information · C. Active Perception Can Improve the Language Interface · D. Scope of Pretrained Semantic Knowledge

Active observation and user clarification provide complementary information for embodied target disambiguation: seeing reveals physical evidence, while asking reveals intent. The framework uses active perception to ground language interaction, while recognizing that current-interaction evidence exceeds pretrained semantic knowledge.

B. Seeing and Asking Provide Different Information

  • Active observation identifies what is physically present, while user clarification specifies what the user actually wants.The two information channels are complementary because visual evidence identifies available targets and user responses identify the intended one.Source: B. Seeing and Asking Provide Different… · p. 5

C. Active Perception Can Improve the Language Interface

  • Fresh views expose package names, colors, stripes, graphics, and other attributes that support more grounded clarification questions.These observations replace fixed-view positional proxies with task-relevant visual descriptions.Source: C. Active Perception Can Improve the… · p. 5
  • Active perception can improve the language interface without eliminating the need for a question.The A6 and A7 traces show that active observation and user interaction can remain sequentially complementary.Source: C. Active Perception Can Improve the… · p. 5
  • Grounded questions let users contribute preferences or intentions, including through free-form answers.A semantically informative user response can still be wasted by a rigid interaction process.Source: C. Active Perception Can Improve the… · p. 5

D. Scope of Pretrained Semantic Knowledge

  • The evaluated tasks require current-interaction evidence that pretrained VLM knowledge alone cannot resolve.Examples include a letter attached to a box, a volume marker on one of two similar books, and whether a medicine box is present in the workspace.Source: D. Scope of Pretrained Semantic… · p. 5
  • A VLM may understand object categories and word meanings while lacking decisive evidence specific to the currently observed scene.The distinction motivates combining visual evidence, user feedback, and prior knowledge during sequential embodied interaction.Source: D. Scope of Pretrained Semantic… · pp. 5–6

E. Consistency Between Physical Evidence and User Feedback · VII. Conclusion

The framework assigns complementary roles to active observation and user clarification in embodied target disambiguation, while acknowledging that conflicts between physical evidence and user feedback remain unaddressed. The robot uses updated visual understanding to continue observing, request clarification, or complete target selection.

E. Consistency Between Physical Evidence and User Feedback

VII. Conclusion

  • The proposed vision-language framework uses active observation as the information backbone for embodied target disambiguation.Source: §VII · p. 8
  • The robot changes viewpoint to acquire physical evidence missing from its current observation.Source: §VII · p. 8
  • Based on updated visual understanding, the robot continues observing, requests user clarification, or completes target selection.Source: §VII · p. 8

Related papers

Loading 2608.13605v1…