Source-linked AI summary
Toward Ambulatory Vision: Learning Visually-Grounded Active View Selection
Juil Koo, Daehyeon Choi, Sangwoo Youn, Phillip Y. Lee, Minhyuk Sung
TL;DR
VLMs excel at static VQA, while active visual perception remains an underexplored part of embodied question answering. VG-AVS trains VLMs to select informative viewpoints using current visual evidence, paired synthetic views, and sequential SFT-then-RL fine-tuning. The approach outperforms zero-shot methods, generalizes to real scenes, and improves existing EQA pipelines.
Problem
VLMs answer questions from static images, while localization and active perception in embodied question answering remain less explored.
Source: §1 · p. 2Method
VG-AVS trains a VLM to choose an informative viewpoint from current visual information, using paired query-target views and sequential SFT-then-RL fine-tuning.
Source: §1 · p. 2Results
Training-based methods significantly outperform zero-shot approaches in VG-AVS and generalize from a small synthetic dataset to real scenes with diverse question types.
Source: §1 · p. 2Takeaways & Limitations
The learned VG-AVS framework can serve as a plug-and-play component that improves existing scene-exploration-based EQA frameworks.
Source: §5 · p. 8SFT relies on annotated actions, and view-level supervision that is available from synthetic paired views is difficult to define or annotate for long-horizon EQA trajectories.
Source: §3.2.1 · p. 5
Abstract
from arXiv · showhide
Vision Language Models (VLMs) excel at visual question answering (VQA) but remain limited to snapshot vision, reasoning from static images. In contrast, embodied agents require ambulatory vision, actively moving to obtain more informative views. We introduce Visually Grounded Active View Selection (VG-AVS), a task that selects the most informative next viewpoint using only the visual information in the current image, without relying on scene memory or external knowledge. To support this task, we construct a synthetic dataset with automatically generated paired query-target views and question-answer prompts. We also propose a framework that fine-tunes pretrained VLMs through supervised fine-tuning (SFT) followed by RL-based policy optimization. Our approach achieves strong question answering performance based on viewpoint selection and generalizes robustly to unseen synthetic and real scenes. Furthermore, incorporating our learned VG-AVS framework into existing scene-exploration-based EQA systems improves downstream question-answering accuracy.
1. Introduction
The introduction reframes VQA as active perception: an agent uses visual clues in its current view to select an informative next viewpoint. It presents VG-AVS, a synthetic training dataset, and sequential SFT-then-RL fine-tuning, reporting gains over zero-shot approaches and generalization to real scenes.
1. Introduction
- VG-AVS targets the underexplored link between partial visual observations and fine-grained viewpoint selection, using only current visual information rather than scene memory or external commonsense knowledge.The introduction contrasts this with snapshot vision and identifies localization and perception as a less-explored EQA component.Source: §1 · p. 2
- The formulation predicts continuous heading rotation, forward translation, and view rotation to select a viewpoint rather than only an agent position.The authors contrast fine-grained control with coarsely discretized actions and multi-turn navigation settings.Source: §1 · p. 2
- The AVS dataset pairs query and target views with questions about information missing from the query view, simulating incomplete visual observations.The dataset is built on ProcTHOR.Source: §1 · p. 2
- Sequential SFT-then-RL fine-tuning first teaches ground-truth transformations, then refines mobility-parameter prediction through unsupervised policy optimization.The paper says combining the strategies improves performance and generalization.Source: §1 · p. 2
- Training-based methods significantly outperform zero-shot approaches in VG-AVS and generalize from a small synthetic dataset to real scenes with diverse question types.The method can also be used as a plug-and-play component to improve existing EQA frameworks.Source: §1 · p. 2
2. Related Work
Prior work studies active VQA in 2D and embodied question answering in 3D, but the cited approaches leave fine-grained active viewpoint selection underexplored. VG-AVS is positioned as a learned policy for selecting views in physically grounded 3D scenes.
2. Related Work
- PixelReasoner, ToA, and Directional Guidance act in 2D image space through cropping, key-frame selection, or coarse direction prediction, remaining within the given frame.The passage contrasts these actions with exploring new viewpoints in the underlying 3D scene.Source: §2 · p. 2
- MindJourney generates observations with a generative model but uses discrete action primitives and beam search rather than a learned policy.VG-AVS instead learns a policy for physically grounded 3D scenes with fine-grained continuous actions.Source: §2 · p. 2
- EQA research has emphasized scene exploration and memory construction, while post-training and action-oriented VLM work has not targeted active perception of missing visual evidence.The passage describes localization and perception as an underexplored EQA component and notes GRPO-based post-training approaches [33, 43, 46].Source: Figure 2 · p. 3
3. Visually-Grounded Active View Selection
VG-AVS frames active perception as learning to move an embodied agent from an insufficient egocentric view to a more informative view for answering a language query. The section develops the AVS training data, continuous-control policy, and staged SFT/RL training.
3.1. AVS Dataset
- The AVS dataset pairs a partially informative query view with an answerable target view and a question that prompts recovery of the missing visual evidence.It is built in ProcTHOR using rendered views and object visibility information.Source: §3.1 · pp. 3–4
3.2. AVS Framework
- The framework models view selection as a continuous policy that maps an egocentric observation and language query to a real-valued action, reaching an answerable view in one step.The action comprises heading rotation, forward translation, and view rotation; height and elevation are fixed for simplicity.Source: §3.2 · p. 4
3.2.1. Supervised Fine-Tuning
- SFT teacher-forces ground-truth actions from the query observation and an action-specific instruction, with paired target views enabling action supervision.This confines SFT to human-annotated actions.Source: §3.2.1 · pp. 4–5
3.2.2. Reinforcement Learning
- RL generates a reasoning sequence followed by an action, then optimizes verifiable answerability and action-format rewards using GRPO.A frozen pretrained VLM checks whether the predicted view supports answering the question.Source: §3.2.2 · p. 5
3.2.3. Bridging SFT and RL
- SFT followed by RL performs better than either objective alone: SFT initializes plausible action magnitudes and directions, while RL refines actions using task-specific rewards.SFT alone quickly saturates, while RL from scratch typically underperforms SFT and is less stable in the continuous action space.Source: §3.2.3 · p. 6
4. Experiments
The experiments show that learned active view selection outperforms zero-shot approaches, transfers to real indoor scenes, and improves Fine-EQA when used as a plug-in. Evaluation covers synthetic and real-scene benchmarks, including question types beyond those used for training.
4.1. Experiments on VG-AVS
- The learned framework consistently outperforms open-source baselines using the same Qwen2.5-VL-7B [1] backbone, while also surpassing larger proprietary models including GPT-5 and Gemini-2.5-Pro.The comparison includes recent spatial VLMs ViLaSR [40] and SpatialReasoner.Source: §4.1 · pp. 6–7
- SFT+RL improves VQA accuracy by more than 5% over either SFT or RL alone and generalizes to unseen counting and state questions.The evaluation includes AVS-ProcTHOR, which covers existence, counting, and state questions.Source: §4.1 · pp. 6–7
- Fine-EQA performs significantly worse on VG-AVS; the authors attribute this to its coarse discrete actions rather than continuous, precise viewpoint adjustments.For action models, predicted actions are used to generate views that the VLM verifier evaluates.Source: §4.1 · pp. 6–7
- 70.70 with SFT+RL versus 64.91 with GPT-5 in average score on AVS-HM3D, despite training only on synthetic existence questions.AVS-HM3D covers real indoor scenes and diverse question types.Source: §4.1 · pp. 7–8
- Qualitative examples show viewpoint refinement revealing partially visible or undersized targets, including a closet in a real indoor scene and EQA views of a laundry room and bar counter.In the bar-counter example, the two-stage method captures the whole counter more clearly than SFT or RL alone.Source: §4.1 · p. 8, §4.2 · p. 8
4.2. Experiments on EQA
- 57.67 versus 52.94 average LLM-Match score when SFT+RL AVS is plugged into Fine-EQA, improving on the base framework.The AVS module refines the terminal view produced by the EQA policy before the VLM verifier answers.Source: §4.2 · p. 8
5. Conclusion
The conclusion frames VQA as active perception through VG-AVS, which selects views using a fine-tuned VLM. The approach reports gains in viewpoint-selection-based question answering, generalization to unseen scenes, and improvements when added to EQA pipelines.
5. Conclusion
- VG-AVS reframes VQA as active perception, using a synthetic dataset and SFT+RL fine-tuning to improve view-selection-based question answering and transfer to unseen scenes and EQA pipelines.The conclusion describes the task as selecting viewpoints from a single image.Source: §5 · p. 8
A.1.1. Comparison with More Training Variants
Additional training comparisons favor the sequential SFT+RL strategy over the tested alternatives. In particular, supervised initialization is more effective than relying on longer SFT or a more elaborate RL reward alone.
A.1.1. Comparison with More Training Variants
- RL with an added distance-based camera-pose reward still fails to learn meaningful actions without SFT warm-up.The reward is inversely proportional to predicted-versus-target pose discrepancy.Source: §A.1.1 · p. 11
- Extending SFT from 7 to 20 epochs yields modest gains but still underperforms the two-stage SFT+RL method.The extended run is intended to vary the training objective while training SFT to convergence.Source: §A.1.1 · p. 11
- Adding Number Token Loss to SFT provides no benefit over vanilla SFT in this setting.The auxiliary loss targets action magnitudes with a regression-aware objective.Source: §A.1.1 · p. 11
A.1.2. More Experiments on EQA
The appendix evaluates adding AVS to the Fine-EQA pipeline on OpenEQA. The plug-in improves average LLM-Match score over the Fine-EQA baseline.
A.1.2. More Experiments on EQA
- The OpenEQA experiment uses the same Fine-EQA configuration as the base EQA pipeline, then adds the AVS framework.Source: §A.1.2 · p. 11
- 6.8 percentage points gain in average LLM-Match score over the Fine-EQA baseline when AVS is added to the pipeline.The results are reported in Table A4.Source: §A.1.2 · p. 11
- The appendix also illustrates the AVS integration into an existing EQA pipeline in Figure A6.Source: §A.1.2 · p. 11
A.1.3. Cross-Dataset Generalization
On the SAT spatial reasoning benchmark, the framework generally improves Qwen2.5-VL-7B despite not being trained for SAT; RL-only is the sole variant that slightly underperforms, and the results indicate no noticeable catastrophic forgetting.
A.1.3. Cross-Dataset Generalization
- The framework generally improves Qwen2.5-VL-7B on SAT spatial reasoning despite not being explicitly trained for the benchmark.SAT includes downstream tasks across synthetic and real-image splits.Source: §A.1.3 · p. 11
- RL-only training is the only evaluated variant that slightly underperforms the backbone on SAT.Source: §A.1.3 · p. 11
- The authors conclude that synthetic-dataset training causes no noticeable catastrophic forgetting and transfers reasonably well to broader spatial reasoning tasks.Source: §A.1.3 · p. 11
A.1.4. Results with Multi-Turn Actions
Multi-turn actions yield no clear performance gains over the single-step setting. The Q+T variant is comparable to Q-only and more stable across rollout lengths, while the results indicate that continuous actions can reach an informative viewpoint in one turn.
A.1.4. Results with Multi-Turn Actions
- Multi-turn actions do not yield clear VQA accuracy gains over the single-step setting.Source: §A.1.4 · p. 11
- SFT+RL (Q+T) is accuracy-comparable to SFT+RL (Q-only) and more stable across single- and multi-turn rollouts, but multi-turn performance does not improve.Source: §A.1.4 · p. 11
- The results indicate that continuous actions enable the agent to reach an informative viewpoint within one turn.Source: §A.1.4 · p. 11
A.2. Training and Experiment Setup Details
This section details VG-AVS training and EQA baseline configurations, benchmark curation, and prompting and qualitative reasoning setups, including an example of the model extending Fine-EQA’s exploration to obtain a view from which the verifier can answer the question.
A.2. Training and Experiment Setup Details
- Training uses Qwen2.5-VL-7B [1] with a frozen vision encoder, 7 SFT epochs followed by 5 GRPO epochs, and a frozen verifier for RL rewards.SFT and RL use batch sizes of 32 and 128, respectively; the reward combines format and verifier terms weighted 0.3 and 1.0, with KL coefficient β = 0.04.Source: §A.2 · p. 11
- The EQA comparison uses Fine-EQA with goal-oriented exploration, prism-dinosiglip+7b, and GPT-5 for region prioritization and termination.The goal-oriented strategy adjusts viewpoints using the current observation and language query.Source: §A.2 · p. 11
A.3.1. AVS Dataset and AVS-ProcTHOR
- AVS training uses ProcTHOR [7] binary existence questions with guaranteed-positive answers, while AVS-ProcTHOR evaluates existence, counting, and state questions, including multiple-choice existence items.The curation pipeline selects target/supporting-object pairs, applies question-specific scene modifications, generates question templates, and retains samples correctly answered from the target view.Source: §A.3.1 · p. 13
- For ProcTHOR viewpoint sampling, target poses come from the ten nearest interactable poses, and a query pose is randomly selected from candidates generated by moving away from the target view.The target pose must satisfy the stated pose rule; query candidates are generated with sampled movement and view-rotation actions and must satisfy their corresponding rule.Source: §A.3.1 · p. 13
A.3.2. AVS-HM3D
- AVS-HM3D [29] contains 208 real-scene samples from Fine-EQA, retaining five locally grounded question types and manually selecting partially informative query views.GPT-5 filters items with unsolvable questions or poor-quality ground-truth views.Source: §A.3.2 · p. 14
A.4. Prompts
- The prompts distinguish direct action prediction in SFT from RL reasoning, which starts with an action guess, reasons, and then revises that action to ease the transition to reason-before-acting.A direct switch to think-then-act after SFT often triggers post-hoc reasoning, motivating the revised RL output format.Source: §A.4 · p. 14
A.5. More Qualitative Comparisons
- Figure A8 provides additional qualitative comparisons on AVS-ProcTHOR and AVS-HM3D, marking correct and incorrect answers by LLM-Match scores.Correct examples have LLM-Match = 5, while incorrect examples have scores ≤2.Source: §A.5 · p. 14, Figure A8 · p. 16
A.6. Qualitative Examples of AVS Framework Reasoning
- Qualitative examples show the model using partial visual cues to infer heading, forward-motion, and view-adjustment actions, then obtaining views answerable by the verifier.Examples cover object, counting, and state questions in AVS-ProcTHOR and AVS-HM3D.Source: Table A7 · p. 17, §A.6 · p. 18, Table A8 · p. 19
- After Fine-EQA [21] provides an observation, our model identifies insufficient visual cues, predicts a <head>, <fwd>, <view> action sequence, and enables the verifier to answer from the updated view.The example illustrates the model reasoning that additional context is needed before acting.Source: Table A9 · p. 21