PaperlayerView exact revision on arXiv

Source-linked AI summary

Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images

Kaname Kitamura, Asako Kanezaki

arXiv:2609.33288v1cs.CVcs.RO

TL;DR

Open-vocabulary EM-EQA uses recorded observations to answer free-form questions, but omnidirectional images bring projection distortion and question-irrelevant content. The paper selects informative, diverse perspective views from those observations; on HM3D OpenEQA, it reports state-of-the-art performance among reported equirectangular results and largely maintains accuracy after reducing observation frames by 65.5%.

  • Problem

    Open-vocabulary EM-EQA with omnidirectional images must contend with equirectangular distortion that degrades VLM recognition and irrelevant visual content that can reduce answer accuracy and increase visual-token burden.

    Source: §1 · p. 2

  • Method

    The method projects omnidirectional observations into cubemap views, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects diverse informative viewpoints.

    Source: §1 · p. 3

  • Results

    65.5% fewer observation frames (170 to 59 per question) after removing rotation views, with accuracy largely maintained; the method also achieves state-of-the-art performance among reported equirectangular results.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    The results suggest that omnidirectional observation histories can provide useful scene coverage for EQA and that the method can extract informative views for a VLM.

    Source: §6 · p. 14

    Evaluation is limited to the HM3D subset of OpenEQA, and the full framework is evaluated only with BLIP-2 as its relevance model.

    Source: §5 · p. 14

Abstract

from arXiv · show

Embodied Question Answering (EQA) requires agents to answer natural language questions about surrounding environments from visual observations. In this work, we focus on open-vocabulary episodic-memory EQA (EM-EQA), where an agent answers free-form questions using recorded observation histories. Omnidirectional images are promising for this task, as they provide wide field-of-view observations that can capture surrounding context without requiring explicit camera rotations. However, omnidirectional images introduce two challenges for EQA: (i) equirectangular projection causes severe geometric distortion that degrades vision-language model (VLM) recognition accuracy, and (ii) feeding equirectangular images directly into VLMs introduces excessive irrelevant background information, reducing answer accuracy and increasing the visual-token burden. To address these challenges, we propose a viewpoint selection method for EM-EQA using omnidirectional images. Our method converts equirectangular observations into perspective views via cubemap projection, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects informative and diverse viewpoints through diversity-aware greedy selection. Experiments on the Habitat-Matterport 3D (HM3D) subset of OpenEQA show that our method achieves state-of-the-art model performance among the reported model results with equirectangular observations. Moreover, after removing rotation views, which reduces observation frames by 65.5%, our method largely maintains its answer accuracy.

1 INTRODUCTION

Open-vocabulary EM-EQA answers free-form questions from recorded histories, and omnidirectional images offer wide coverage. Equirectangular distortion and irrelevant visual content, however, challenge VLM recognition and answer generation.

1 INTRODUCTION

  • Omnidirectional observations provide wide field-of-view coverage at visited locations and may reduce the need for explicit rotation actions.Source: §1 · p. 2
  • Equirectangular projection distorts objects, especially near the poles, while omnidirectional images also include question-irrelevant content that can distract VLMs and increase visual-token burden.Source: §1 · p. 2
  • The paper investigates open-vocabulary EM-EQA with omnidirectional images, a setting described as largely underexplored in prior EQA research.Source: §1 · p. 3
  • The method converts equirectangular observations into cubemap-based perspective views, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects diverse viewpoints.Source: §1 · p. 3
  • 65.5% fewer observation frames (170 to 59 per question) after removing rotation views, with accuracy largely maintained; the method also achieves state-of-the-art performance among reported equirectangular results.Source: §1 · p. 3

2 RELATED WORK

Prior EQA work largely assumes perspective observations or explicit 3D representations, while omnidirectional EM-EQA combines recorded histories with wide-field observations. The method formulates viewpoint selection around question relevance and diversity, then uses cubemap views, fine-tuned BLIP-2 scoring, and diversity-aware selection.

2 RELATED WORK

  • Prior EQA methods mainly assume perspective observations or explicit 3D representations, distinguishing this work’s selection of perspective views from omnidirectional histories for EM-EQA.Related work also describes retrieval from frames, captions, or structured memories, and omnidirectional QA without embodied navigation.Source: §2 · pp. 3–4

3.1 Problem Formulation

  • The EM-EQA objective selects K viewpoints from cubemap-derived candidates to maximize the ideal probability of generating the correct answer given the question.The formulation motivates both question relevance and reduced redundancy under a limited visual-input budget.Source: §3.1 · p. 5

3.2 Cubemap Projection

  • Cubemap projection converts each equirectangular observation into six reduced-distortion perspective views, producing 6T candidate viewpoints from T observations.The six faces are Front, Back, Left, Right, Top, and Bottom.Source: §3.2 · pp. 5–6

3.3 Fine-tuned BLIP-2 for Relevance Estimation

  • BLIP-2’s ITM head scores image-question relevance; the method fine-tunes it on VQA data, using supervised contrastive learning with identical question texts as positives.LoRA adapts selected Q-Former layers while the image encoder and learned queries remain frozen and the ITM head is fully fine-tuned.Source: §3.3 · pp. 6–7

3.4 Diversity-Aware Greedy Selection

  • Diversity-Aware Greedy Selection prioritizes high-confidence candidates by relevance score and accepts a candidate only when its maximum similarity to selected views is below a threshold.It fills remaining slots from rejected high-confidence candidates, then low-confidence candidates; with no high-confidence candidates, it uses uniform temporal sampling.Source: §3.4 · pp. 7–8

4 EXPERIMENTS

Experiments on OpenEQA’s HM3D subset compare viewpoint selection across observation types and rotation settings. The method leads among evaluated strategies, and removing rotation views sharply reduces stored frames while largely preserving equirectangular answer accuracy.

4.1 Experimental Setup

  • Evaluation covers all 557 OpenEQA HM3D questions across seven categories, using LLM-Match to compare selection against temporal sampling and BLIP retrieval on omnidirectional and perspective histories, including rotation-removed histories.The rotation-removed setting uses the same trajectories but omits views from rotation actions.Source: §4.1 · pp. 8–9

4.2 Results and Analysis

  • Our method scores highest among evaluated selection strategies across observation types and rotation settings; removing rotation views drops equirectangular scores 2.0 points (57.9 to 55.9), versus 15.6 for standard perspective (61.1 to 45.5).The comparison uses maximum LLM-Match scores over K.Source: §4.2 · pp. 9–10, Table 2 · p. 10
  • Our method achieves the best or tied-best score in five of seven categories, with Spatial Understanding at 55.8% versus 39.1% for Multi-Frame VLMs and 37.7% for RAG with BLIP; it also outperforms both baselines with three other answer models.The category comparison is at K = 5 on equirectangular observations in the Original setting; the alternative answer models are GPT-5.6 Sol, InternVL3-78B, and Llama 4 Scout.Source: Table 4 · p. 11, §4.2 · p. 11
  • Across observation settings, scores often plateau around K = 5–8, and increasing K further provides limited benefit while potentially adding redundant views and computational cost.The method also outperforms compared methods under the same input-view budget across the tested K values.Source: §4.2 · p. 11

4.3 Ablation Studies

  • At K = 5, fine-tuning raises cubemap scoring from 41.7% to 55.2%, and adding Diversity-Aware Greedy Selection raises it to 57.5%; that selector also beats MMR (at most 55.8%) and R-EQA-style selection (50.3%).The ablation reports τ = 0.97 as the best similarity threshold; all tested γ values from 0.1 to 0.5 improve over γ = 0.0, and γ = 0.1 is adopted.Source: §4.3 · pp. 11–12, Table 6 · p. 12

4.4 Qualitative Analysis

  • Qualitative examples show fine-tuning finding target-containing views where pretrained BLIP-2 selects irrelevant ceiling frames, while diversity filtering avoids repeated partial views and preserves spatial relationships.In the bed example, diversity-aware selection captures the relationship between the bed and target object, supporting the correct answer.Source: §4.4 · pp. 12–13, Figure 4 · p. 13

4.5 Inference Time Analysis

  • Removing explicit rotation views cuts equirectangular inference time from 27.76 to 12.10 seconds as stored frames fall from 170 to 59; viewpoint selection adds computation, and its current implementation is unoptimized.The authors note that software optimization cannot remove the physical rotation burden of collecting explicit rotation views.Source: Table 7 · p. 13, §4.5 · pp. 13–14

5 LIMITATIONS

The method has limits in scene-level and multi-view reasoning, cubemap view selection, computational efficiency, and evaluation scope.

5 LIMITATIONS

  • The selected views may be insufficient for global scene understanding, trajectory reasoning, multi-view integration, absence verification, counting, and multi-instance disambiguation.The authors suggest structured memory as a way to maintain object identities, temporal context, and scene-level evidence for these cases.Source: §5 · p. 14
  • Cubemap faces can split or truncate boundary objects, and six fixed directions may miss question-optimal continuous views.The authors call for more adaptive projection strategies.Source: §5 · p. 14
  • Cubemap candidate expansion, BLIP-2 scoring, and image-feature extraction add computation despite improved answer accuracy.The authors note that a more efficient implementation could reduce the overhead.Source: §5 · p. 14
  • Evaluation is limited to the HM3D subset of OpenEQA, with the full selection framework tested only using BLIP-2 relevance.Other relevance models, environments, and real robots with omnidirectional cameras remain future work.Source: §5 · p. 14

6 CONCLUSION

The method combines cubemap projection, question-conditioned relevance, and diversity-aware selection for omnidirectional EM-EQA. On HM3D OpenEQA, it reaches state-of-the-art performance among reported models using equirectangular observations and largely maintains accuracy after rotation views are removed.

6 CONCLUSION

  • On HM3D OpenEQA, removing rotation views cuts observation frames by 65.5% while largely maintaining answer accuracy.The method also achieves state-of-the-art model performance among reported results with equirectangular observations.Source: §6 · p. 14

Supplementary Material

The supplementary material provides additional analyses and implementation details, with default evaluation settings specified for its results.

Supplementary Material

  • Supplementary results use 557 HM3D OpenEQA questions, Qwen2-VL-72B-Instruct, K = 5, γ = 0.1, τ = 0.97, and LLM-Match scores from gpt-4-turbo-2024-04-09 unless stated otherwise.Tables and algorithms without an “S” prefix refer to the main paper.Source: Supplementary Material · p. 18

S1 Details of Additional Comparisons

The supplementary comparison details describe MMR and an R-EQA-style selector, and specify controls for answer-model comparisons.

S1 Details of Additional Comparisons

  • MMR uses the same cubemap candidates, fine-tuned BLIP-2 relevance, and BLIP-2 image embeddings, selecting views iteratively.The highest reported score is 55.8% at λ = 0.5.Source: S1 Details of Additional Comparisons · p. 18
  • The R-EQA-style baseline captions cubemap candidates with Qwen2.5-VL-7B-Instruct, embeds captions and the question with Sentence-BERT, and passes the top five views to the answer model.It follows the publicly described R-EQA retrieval policy and is not a full reproduction.Source: S1 Details of Additional Comparisons · p. 18
  • For Tab. 5, answer models receive identical precomputed selections, prompts, and a 128-token generation limit.The passage also specifies model-specific inference settings.Source: S1 Details of Additional Comparisons · p. 18

S2 Qualitative Examples

The examples show successful implicit target inference and a route-following failure. The failure reflects selection based on view relevance and redundancy without modeling observation order or room connectivity.

S2 Qualitative Examples

  • Figure S1 presents views selected by the method for two qualitative examples.Source: Figure S1 · p. 19
  • The route question is answered incorrectly: the method selects individually relevant views but does not model temporal order or room connectivity.The ground truth is “The restroom,” while the method answers “the storage room” (score 1).Source: S2 Qualitative Examples · p. 19
  • For an exercise question that does not name its target, ITM-based selection retrieves stationary-bike views and the answer model returns “exercise bike” (score 5).This matches the ground truth “The Stationary Bike”; Multi-Frame VLMs score 4 and RAG with BLIP scores 5.Source: S2 Qualitative Examples · pp. 18–19

S3 Fallback Analysis

The fallback analysis reports fallback frequencies and LLM-Match, and finds that replacing uniform sampling for questions with no candidate above γ lowers overall LLM-Match.

S3 Fallback Analysis

  • Table S2 reports each fallback mechanism’s frequency and the LLM-Match of questions triggering it versus the remaining questions.Source: S3 Fallback Analysis · p. 19, Table S2 · p. 20
  • In Table S2, Chigh and Clow are candidates with ITM scores at least γ and below γ, respectively.Source: Table S2 · p. 20
  • 57.5% to 56.1%: overall LLM-Match decreases when uniform sampling is replaced by relevance- and diversity-based selection for questions with no candidate above γ.Source: S3 Fallback Analysis · pp. 19–20

S4 Relevance Model Comparison

The comparison uses a threshold-free Top-K protocol with shared cubemap candidates and answer model, varying the relevance model without diversity-aware greedy selection.

S4 Relevance Model Comparison

S5 Observation Units and Runtime Breakdown

Each omnidirectional observation produces six cubemap candidates; the runtime breakdown profiles selection components and answer generation under a specified GPU and batch setup.

S5 Observation Units and Runtime Breakdown

Related papers

Loading 2609.33288v1…