PaperlayerView exact revision on arXiv

Source-linked AI summary

Explore until Confident: Efficient Exploration for Embodied Question Answering

Allen Z. Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, Dorsa Sadigh

arXiv:2403.15941v3cs.ROcs.AIcs.CVcs.LG

TL;DR

EQA asks a robot to explore diverse 3D scenes to answer questions, but VLM-based approaches face challenges in this setting. This paper combines external semantic mapping with uncertainty quantification, and reports more efficient exploration in simulated and hardware experiments. The authors also identify omitted view orientation and inconsistent spatial reasoning in current open-source VLMs as limitations.

  • Problem

    EQA requires a robot to explore a diverse 3D scene without an a-priori map and decide when it can answer the question.

    Source: §I · p. 1

  • Method

    The framework combines an external semantic map for exploration with uncertainty quantification of VLM answers to guide stopping.

    Source: §I · p. 1

  • Results

    Simulated and hardware experiments show improved exploration efficiency, achieving similar success levels with fewer steps.

    Source: §VIII · p. 12

  • Takeaways & Limitations

    The authors expect performance to improve with more powerful VLMs that have stronger spatial and semantic reasoning, and describe their method as complementary to them.

    Source: §VIII · p. 12

    The setup omits view orientation from semantic-value estimation, and performance is limited by inconsistent spatial reasoning in current open-source VLMs.

    Source: §VIII · p. 12

Abstract

from arXiv · show

We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about the answer to a question. In this work, we leverage the strong semantic reasoning capabilities of large vision-language models (VLMs) to efficiently explore and answer such questions. However, there are two main challenges when using VLMs in EQA: they do not have an internal memory for mapping the scene to be able to plan how to explore over time, and their confidence can be miscalibrated and can cause the robot to prematurely stop exploration or over-explore. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM - leveraging its vast knowledge of relevant regions of the scene for exploration. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration - leading to a more calibrated and efficient exploration strategy. To test our framework in simulation, we also contribute a new EQA dataset with diverse, realistic human-robot scenarios and scenes built upon the Habitat-Matterport 3D Research Dataset (HM3D). Both simulated and real robot experiments show our proposed approach improves the performance and efficiency over baselines that do no leverage VLM for exploration or do not calibrate its confidence. Webpage with experiment videos and code: https://explore-eqa.github.io/

I. INTRODUCTION

The introduction frames EQA as exploration in diverse 3D scenes and identifies missing VLM scene memory and miscalibrated confidence as challenges. It proposes external semantic mapping and multi-step conformal prediction, then situates the approach among work on EQA, embodied VLMs, semantic mapping, and calibration.

I. INTRODUCTION

  • VLM-based EQA faces absent scene memory and miscalibrated confidence; the framework addresses these with external semantic mapping and uncertainty quantification.The authors connect miscalibration to difficulty deciding when to stop exploring.Source: §I · p. 1
  • The framework uses VLM reasoning to prioritize relevant regions and multi-step conformal prediction to calibrate answer uncertainty and guide stopping.The robot updates possible answers from new views and stops when one answer remains; conformal prediction is described as supporting coverage and limiting set size.Source: §I · pp. 1–2
  • Simulated and hardware experiments are reported to improve EQA efficiency over baselines lacking VLM semantic exploration or calibrated stopping.The simulated evaluation uses HM-EQA, a dataset based on realistic human-robot scenarios and HM3D scans [12].Source: §I · p. 2

II. RELATED WORK

  • Earlier EQA methods trained exploration and answering from scratch on largely synthetic scenes, while newer foundation-model approaches assume full scene information and omit exploration.The cited earlier EQA work considered simple questions such as object color [3] [4] [5], whereas the newer work addresses similar question types [14] [15] [16] [17].Source: §II · p. 2
  • Related embodied VLM work uses view selection or search heuristics, while this framework combines uncertainty estimation with active exploration in larger scenes.The cited examples include informative viewing angles [8] and LLM-generated search heuristics [25].Source: §II · p. 2
  • Unlike prior semantic-mapping approaches that build maps offline, this method builds one online and uses VLM probabilistic outputs without extra training or another foundation model.For stopping, it applies multi-step conformal prediction to EQA rather than using it to decide when to ask a human [11].Source: §II · p. 3

III. PROBLEM FORMULATION

The problem is modeled as answering multiple-choice questions in previously unknown scenarios while minimizing exploration steps. The framework combines VLM predictions with an external map and frontier-based navigation to pursue that goal.

III. PROBLEM FORMULATION

  • Each EQA scenario is a tuple containing a 3D scene, time horizon, initial robot pose, question, and ground-truth answer.The scenario is sampled from an unknown distribution; the question and answer complete the tuple definition.Source: §III · p. 3
  • The robot receives RGB and depth images at stops, uses a collision-free planner limited to 3 m per move, and assumes perfect simulated odometry.In real-world settings, pose updates use a localization algorithm.Source: §III · p. 3
  • At each step, the VLM returns softmax scores over the four multiple-choice labels from the current RGB image and question.The formulation denotes the input as x_t and the prediction vector as f-hat(x_t).Source: §III · p. 3
  • The objective is to answer correctly in unseen scenarios using as few time steps as possible, without over-exploration.Source: §III · p. 3

A. Overview

  • The framework queries answer probabilities, question-image relevance, and semantic values, stores them with occupancy and unknown regions in an external map, then explores and decides when to stop.Stopping is based on answer prediction and question-image relevance.Source: A. Overview · p. 4

B. Exploration Map and Frontier-Based Exploration

  • Depth-based TSDF fusion updates voxel occupancy and explored status before projecting the 3D representation into a 2D point map.The map grows in length and width as exploration proceeds; its height is fixed at 3.5 m.Source: B. Exploration Map and Frontier-Based… · p. 4
  • Frontier-based exploration identifies boundaries between explored and unexplored regions, samples a destination, and faces the robot toward the unexplored side.The planner is a heuristic-based 2D strategy built on the occupancy and exploration map.Source: B. Exploration Map and Frontier-Based… · p. 4

C. VLM Visual Prompting for Semantic Value (Fig. 3)

Visual prompting assigns semantic values to possible exploration locations using local point comparisons and a global view-level query. The combined value is stored in the semantic map to weight frontier sampling.

C. VLM Visual Prompting for Semantic Value (Fig. 3)

  • The method samples three points from visible free space, projects them onto the image, and labels them for VLM prompting.Farthest point sampling is used to cover the visible points; the authors report that three points suffice for distinct image regions.Source: C. VLM Visual Prompting for Semantic… · p. 4
  • The VLM’s normalized probabilities over labeled directions define the local semantic value, which compares points within one image rather than across views.The method also prompts the model to choose which direction in the image it would explore.Source: C. VLM Visual Prompting for Semantic… · p. 4
  • A separate yes-or-no prompt asks whether any direction in the image is worth exploring, with the normalized Yes probability defining the global semantic value.Source: C. VLM Visual Prompting for Semantic… · p. 4
  • Temperature-scaled local and global values are combined into an overall semantic value and saved in the map to weight frontier sampling.Gaussian smoothing spreads each value around its point to support exploration.Source: C. VLM Visual Prompting for Semantic… · p. 5, Figure 3 · p. 5

D. Semantic-value-weighted Frontier Exploration

The exploration policy weights frontier sampling by semantic value around relevant regions. For stopping, multi-step conformal prediction intersects answer sets over time, while raw VLM confidence criteria can lead to under- or over-exploration.

D. Semantic-value-weighted Frontier Exploration

  • Frontier sampling uses semantic values at candidate points and nearby points in the normal direction, with Gaussian smoothing around prompted points.The nearby-point value can guide the robot toward relevant regions that are not close to its current pose.Source: D. Semantic-value-weighted Frontier… · p. 5

V. STOPPING CRITERION FOR EXPLORATION AND ANSWERING THE QUESTION

  • Experiments found that raw-softmax stopping criteria can lead the robot to under-explore or over-explore in many scenarios.The compared raw-score criteria include answer entropy and a question-image relevance score.Source: §V · pp. 5–6
  • Multi-step conformal prediction maintains intersected answer sets and stops when only one answer remains, with a user-specified probability of coverage.The method uses a calibrated threshold and minimizes prediction-set size to support stopping as soon as possible, assuming calibration and test scenarios share a distribution.Source: §V · p. 6, Figure 4 · p. 6

A. Background: Conformal Prediction

Conformal prediction uses a calibration dataset to form prediction sets for new inputs. Its coverage threshold is user-defined, and the method uses relevance-weighted confidence scores to calibrate the sets.

A. Background: Conformal Prediction

  • Given calibration pairs drawn i.i.d. from an unknown distribution, conformal prediction forms a set of answers for a new input and label.The set contains the true label with high probability.Source: A. Background: Conformal Prediction · p. 6
  • The user-defined threshold 1−ϵ controls the size of the prediction set.Source: A. Background: Conformal Prediction · p. 6
  • Calibration uses relevance-weighted confidence scores that are high when the VLM is confident and the image is relevant, then includes labels above a calibrated threshold.The threshold is obtained from the empirical quantile of calibration-set nonconformity scores.Source: A. Background: Conformal Prediction · p. 6

B. Applying Multi-Step CP for Embodied Question Answering

The multi-step method calibrates confidence over episode-level input sequences, then constructs prediction sets causally as observations arrive. Intersecting these sets preserves the sequence-level coverage guarantee and supplies a stopping rule.

B. Applying Multi-Step CP for Embodied Question Answering

  • Episode-level calibration uses input sequences collected under the exploration policy and a nonconformity score based on sequence-level relevance-weighted confidence.The calibrated threshold defines a sequence-level prediction set guaranteed to contain the ground-truth answer with probability 1−ϵ.Source: B. Applying Multi-Step CP for Embodied… · p. 6
  • At each time step, the robot constructs a prediction set using observations available so far, rather than requiring the full test sequence in advance.This causal construction is the basis for applying the sequence-level guarantee during exploration.Source: B. Applying Multi-Step CP for Embodied… · pp. 6–7
  • For test scenarios drawn from the modeled distribution, the intersection of causally constructed sets contains the ground-truth answer with probability 1−ϵ.The causally constructed sets contain the sequence-level set, and their intersection equals that sequence-level set.Source: B. Applying Multi-Step CP for Embodied… · p. 7
  • The robot halts when the intersected sets contain one answer; if the time horizon ends with multiple or no answers, it selects the highest-scoring answer from the most relevant time step.Source: B. Applying Multi-Step CP for Embodied… · p. 7

VI. HM-EQA DATASET

HM-EQA brings EQA evaluation to diverse, realistic indoor scenes with more open-ended questions than prior synthetic-scene benchmarks. The paper also describes a multiple-choice extension for questions without supplied choices and outlines experiments on exploration and stopping.

VI. HM-EQA DATASET

  • HM-EQA is built on HM3D to study more realistic, diverse scenes and open-ended questions that may require semantic reasoning.The paper contrasts this scope with prior work focused on synthetic scenes and simpler questions about basic attributes of large furniture.Source: §VI · p. 7
  • GPT4-V generated questions from twelve sampled views per scene, followed by manual removal of overly simple or hallucinated questions; the dataset contains 500 questions from 267 scenes.Source: §VI · p. 7
  • The questions cover identification, counting, existence, state, and location, and options are padded to four when needed.The reported category shares are 16.2%, 16.0%, 21.6%, 20.2%, and 26.0%, respectively.Source: §VI · p. 7

A. Extension: Answering Questions without Multiple Choices

VII. EXPERIMENTS AND DISCUSSION

  • The experiments ask whether semantic exploration and calibrated stopping reduce steps while maintaining EQA success against their respective baselines.Source: §VII · p. 8

A. Implementation Details

The experiments compare VLM-guided semantic exploration with frontier and CLIP-based baselines, examine the roles of global and local semantic values, and test temperature scaling. Results show improved early exploration efficiency, while excessive or insufficient semantic weighting reduces efficiency and VLM answer errors constrain success.

A. Implementation Details

  • Experiments use Prismatic VLM because probability outputs are required, and hardware trials deploy the framework on a Fetch robot in six home/office-like scenarios.Simulated HM-EQA scenarios run in Habitat; an iPhone 12 provides wide-angle RGB images on the robot.Source: A. Implementation Details · p. 8

B. Q1: Semantic Exploration - Baselines

  • Baselines include frontier-only FBE, CLIP-semantic CLIP-FBE, and variants of the method without GSV or LSV.CLIP-FBE uses multi-scale relevancy for fine-grained localization, while the ablations isolate global and local semantic values.Source: B. Q1: Semantic Exploration - Baselines · p. 8

C. Q1: Semantic Exploration - Simulation Results

  • Compared with FBE and CLIP-FBE, the method achieves success with fewer steps, with the difference becoming significant around 20%–30% of maximum allowable steps.The comparison measures average answer success across varying normalized time steps.Source: C. Q1: Semantic Exploration -… · p. 9
  • Removing either GSV or LSV reduces performance to around FBE levels or worse, while LSV-only exploration can initially succeed faster but later stagnates.Without GSV, the planner can be myopic; without LSV, semantic values are less fine-grained and exploration is less efficient.Source: C. Q1: Semantic Exploration -… · p. 9
  • Both overly high and overly low semantic-value temperature scalings reduce efficiency; high scaling speeds early exploration but worsens later performance.The authors suggest that high scaling can over-prioritize regions when VLM reasoning is unreliable.Source: C. Q1: Semantic Exploration -… · p. 9
  • Overall success remains around 60% even at the maximum time steps, largely because the VLM can answer incorrectly after seeing the relevant object.The authors identify improved VLM question answering as a potential way to reduce these cases.Source: C. Q1: Semantic Exploration -… · p. 9

D. Q1: Semantic Exploration - Comparing to CLIP-FBE

The comparison illustrates how VLM reasoning can identify a route to a relevant room when CLIP relevancy offers little guidance. Fine-tuning improves question-answering success, while the stopping baselines use answer entropy or question-image relevance thresholds.

D. Q1: Semantic Exploration - Comparing to CLIP-FBE

  • The VLM identifies the bedroom door as an exploration route in a partial living-room view, whereas CLIP relevancy can offer little signal when the place of interest is not visible.The example concerns finding a bedroom curtain; CLIP attention instead falls on the sofa, even when prompted with “Bedroom.”Source: D. Q1: Semantic Exploration -… · p. 9, Figure 8 · p. 10
  • CLIP-FBE improves exploration efficiency but offers limited semantic reasoning, while VLMs can also over-prioritize irrelevant areas when over-confident.The paper attributes CLIP's limitation to bag-of-words behavior and notes that excessive semantic exploration can worsen performance.Source: D. Q1: Semantic Exploration -… · p. 9

E. Q1: Semantic Exploration - Fine-tuning VLM

  • With the same Prismatic VLM, fine-tuning on 300 scenarios raises success rate to 68.1% from 56.2% on 200 test scenarios.The authors report that dataset fine-tuning can better ground question answering in the scenario distribution and EQA tasks.Source: E. Q1: Semantic Exploration -… · p. 9

F. Q2: Stopping Criterion - Baselines

  • The stopping baselines use answer entropy or question-image relevance thresholds, and evaluation tests all methods on the same 200 scenarios after a 300/200 calibration-test split.Entropy stopping selects the highest-probability answer at stopping; relevance stopping uses the question-image relevance score.Source: F. Q2: Stopping Criterion - Baselines · p. 10

G. Q2: Stopping Criterion - Simulation Results

Simulation results show that conformal prediction (CP) improves stopping efficiency over uncalibrated baselines, and hardware experiments report similar success to Relevance with fewer steps. The conclusion describes the framework’s overall gain as achieving similar success with fewer steps.

G. Q2: Stopping Criterion - Simulation Results

  • Entropy stopping significantly underperforms because irrelevant views can elicit confident, biased answers with low prediction entropy, causing premature stops.The evaluation varies thresholds for CP, Entropy, and Relevance and compares normalized steps against success rate.Source: G. Q2: Stopping Criterion - Simulation… · p. 10, Figure 9 · p. 10
  • At 58% success, our method uses about 71% of maximum steps, compared with 85% for Relevance, while CP calibrates confidence to improve efficiency.The authors also report that CP offers the most efficiency improvement when the desired success rate is high [11].Source: G. Q2: Stopping Criterion - Simulation… · p. 10

H. Q2: Stopping Criterion - Hardware Results

VIII. CONCLUSION

  • The conclusion characterizes the framework’s simulated and hardware gains as similar success levels achieved with fewer exploration steps.The framework combines VLM reasoning about relevant locations with confidence calibration for stopping.Source: §VIII · p. 12

APPENDIX A PROOF OF CLAIM 1

The appendix proof states that it follows Ren et al. [11, Claim 1], then describes semantic-map implementation details and how candidate questions and answers are generated for HM-EQA.

APPENDIX A PROOF OF CLAIM 1

  • The proof of Claim 1 follows Ren et al. [11, Claim 1].The supplied passage gives no further proof steps.Source: Appendix A · p. 15

A. Semantic map

  • Frontier sampling weights combine semantic value at the frontier with the average semantic value 3 m ahead in the unexplored region, using temperature scaling values of 1.The authors say balancing these values helps guide exploration toward relevant regions; online adaptation may further improve efficiency.Source: A. Semantic map · p. 15, Figure A1 · p. 15

B. HM-EQA Dataset

  • GPT4-V generates HM-EQA candidate questions and answers from twelve randomly sampled scene views, using a prompt and three manually written examples.The appendix shows the prompt and example views, including a question generated from one view.Source: B. HM-EQA Dataset · p. 15, Figure A2 · p. 15, Figure A3 · p. 16

APPENDIX C ADDITIONAL EXPERIMENT RESULTS

Additional experiments report higher success with newer VLMs, unreliable raw relevance scores, and variation in semantic-exploration gains across question categories. They also examine scene-size robustness and a view-orientation heuristic.

EQA results using other VLMs

  • Without fine-tuning, LLaVA 1.6 reaches 65.3% success and official GPT4-V reaches 73.9%, versus 58.4% in the main experiments at maximum normalized time.The authors attribute these results to improved question answering and exploration capabilities of the VLMs.Source: Appendix C · p. 16
  • The authors identify inconsistent spatial reasoning by current open-source VLMs across scenarios as a performance constraint.They suggest training on simulated or egocentric exploration data as one possible direction for addressing it.Source: Appendix C · p. 16

Miscalibrated question-image relevance score

  • Most successful scenarios have a highest question-image relevance score below 0.5, so the raw score alone is not a reliable confidence indicator.The scores span a wide distribution even when the answer at the highest-scoring step is correct.Source: Figure A5 · p. 16, Appendix C · p. 16

Semantic exploration results for each question category

  • Semantic exploration improves most over baselines for Count and Existence questions, while it shows no improvement for Identification and State questions remain difficult.Count and Existence often indicate a location of interest; State questions can involve small objects requiring close views, while Identification requires stronger semantic reasoning to infer exploration directions.Source: Appendix C · p. 17, Figure A6 · p. 17

Semantic exploration results for varying scene sizes

  • Across scenes from 150 to 750 m2, success rate is roughly invariant to scene size; the spike near 600 m2 reflects few scenarios in that bin.The authors take this as evidence that time-step normalization is reasonable and frontier-based exploration is essential across methods.Source: Figure A7 · p. 17, Appendix C · p. 17

Improving semantic exploration by considering view orientation

  • Adding a bias toward previously observed view orientations raises success from 58.4% to 59.3%.The current semantic-value setup does not consider view orientation, which can matter when occlusion makes an object visible only from a particular direction.Source: Appendix C · p. 17

Related papers

Loading 2403.15941v3…