PaperlayerView exact revision on arXiv

Source-linked AI summary

Rendering-Free Lookahead for Question-Guided Active Vision

Koya Sakamoto, Daichi Azuma, Shuhei Kurita, Naoya Chiba, Yusuke Iwasawa, Yutaka Matsuo, Taiki Miyanishi

arXiv:2610.11039v1cs.CVcs.RO

TL;DR

Viewpoint-dependent question answering requires camera motions that reveal evidence hidden from the current view, but selecting them means anticipating the value of unseen views. RFL distills answerability scores for rendered future views into a policy that predicts candidate-motion values without deployment rendering. On 377 E3VS-Bench episodes in unseen scenes, it improves mean judge score by 43% over the same VLM acting directly.

  • Problem

    In unseen scenes without a map or target viewpoint, an agent must infer which camera motion will reveal evidence needed to answer a question, while demonstrations alone do not quantify the usefulness of alternative motions.

    Source: §I · p. 1

  • Method

    RFL trains a privileged teacher to score rendered future views with a frozen VLM, then distills short-horizon answerability values into a student that scores candidate motions without deployment rendering.

    Source: §I · pp. 1–2, §VI · p. 7

  • Results

    RFL raises mean judge score from 2.14 to 3.05 (+43%) over the same VLM acting directly on 377 E3VS-Bench episodes across 21 unseen scenes.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support learning camera-control policies for viewpoint-dependent question answering from privileged visual lookahead transferred into deployment-time action values.

    Source: §I · p. 2, §VI · p. 7

    RFL requires counterfactual rendering at training time, although the discussion says any free-viewpoint renderer can serve.

    Source: G. Discussion and Limitations · p. 7

Abstract

from arXiv · show

Active robot vision requires controlling the camera to reveal task-relevant information that is hidden from the current viewpoint. For example, determining what is inside a box may require raising the camera and looking down into it. For viewpoint-dependent question answering, the challenge is to select camera motions that expose the visual evidence needed to answer the question. Although vision-language models (VLMs) can interpret observed images, selecting such motions requires anticipating the usefulness of unseen views. We quantify this usefulness as answerability, a VLM's estimate that a view suffices to answer the question, and present Rendering-Free Lookahead (RFL), a viewpoint-selection policy that ranks candidate camera motions by predicted future answerability. RFL transfers visual lookahead from deployment to offline training. At training, a privileged teacher renders candidate future views in 3D Gaussian Splatting (3DGS) scenes and uses a frozen VLM to compute one- and two-step answerability targets. Through two-stage distillation, a student learns to predict these action values from the question, recent visual observations, and a candidate camera motion. At deployment, RFL uses these predicted values to select camera motions without rendering future views. On 377 E3VS-Bench test episodes in unseen environments, RFL improves the mean judge score by 43\% over a direct-action baseline using the same VLM. These results support learning camera-control policies from privileged visual lookahead for viewpoint-dependent question answering.

I. INTRODUCTION

RFL addresses viewpoint-dependent question answering by learning which camera motions are likely to reveal missing evidence. It transfers privileged visual lookahead from training into a policy that selects motions without rendering future views at deployment.

I. INTRODUCTION

  • In E3VS, an agent with no target viewpoint or scene map must use egocentric RGB observations to find evidence in an unseen scene within a limited action budget.Training scenes permit arbitrary-view rendering, but the deployment scene is unknown.Source: §I · p. 1
  • A privileged teacher scores rendered future views with a frozen VLM, and a student learns one- and two-step answerability targets to select motions without deployment rendering.The targets cover candidate motions, including alternatives not executed by the teacher.Source: §I · pp. 1–2
  • RFL raises mean judge score from 2.14 to 3.05 (+43%) over the same VLM acting directly on 377 E3VS-Bench episodes across 21 unseen scenes.It also surpasses GPT-5.1 (2.81) and teacher-action cloning (2.47).Source: §I · p. 2
  • The simulation-trained policy achieves the highest mean judge score among compared methods on a real 6-DoF arm across three viewpoint-dependent questions, without test-time rendering.The introduction also reports improvements over the same VLM acting directly and cloning teacher actions on unseen scenes.Source: §I · p. 2

II. RELATED WORK

Prior active vision and question-answering work uses questions to guide exploration or answerability to guide stopping and location estimates. RFL instead scores candidate future viewpoints for question-conditioned camera control, using future-view supervision only during training.

A. Active Vision and Embodied Question Answering

  • Active EQA uses natural-language questions to guide evidence gathering, while prior answerability methods guide stopping or estimate answerable scene locations rather than select camera motions.RFL predicts answerability at candidate future viewpoints and uses it as the action value.Source: A. Active Vision and Embodied Question… · p. 2
  • VG-AVS learns viewpoint adjustments with correctness rewards, whereas RFL scores candidate motions using graded, question-conditioned short-horizon targets, including unexecuted alternatives.Planning with the Views instead uses a target image and top-down reference; E3VS supplies neither a scene map nor target viewpoint.Source: A. Active Vision and Embodied Question… · p. 2

B. Visual Lookahead for Action Selection

  • RFL uses rendered future views during training to distill question-conditioned action values that the deployed policy predicts directly, without rendering or generating future observations.This transfers future-view utility into the policy rather than requiring future prediction at deployment.Source: B. Visual Lookahead for Action Selection · p. 2

C. Privileged Supervision and Value Distillation

RFL uses privileged future-view supervision to learn candidate-motion values for a question-answering agent that has only egocentric observations in an unseen scene. Its deployed policy scores candidate motions from the question, recent observations, and action.

C. Privileged Supervision and Value Distillation

  • RFL constructs graded, question-conditioned answerability targets by scoring rendered future views, supervising every candidate camera motion at sampled training states, including unexecuted alternatives.This differs from supervision limited to selected actions or action preferences.Source: C. Privileged Supervision and Value… · p. 2

III. PROBLEM STATEMENT

  • In E3VS, the agent receives a question and egocentric RGB observations but no scene map or target viewpoint; its camera actions comprise ten discrete motions and stop.The actions include six translations and four rotations, and a colliding motion consumes a step without changing pose.Source: §III · p. 2

IV. METHOD

  • An offline teacher renders and scores future views to create one- and two-step targets, while the student learns to predict them from the question, recent observations, and candidate motion.At deployment, the policy executes the highest-scoring candidate and repeats with new observations, without rendering future views.Source: Figure 2 · p. 3, §IV · p. 3

A. Answerability Probe

The section defines answerability as a VLM estimate of whether a view can answer a question, then uses rendered future views to build short-horizon teacher targets for candidate camera motions.

A. Answerability Probe

  • The probe prompts a frozen VLM to judge whether an observation contains enough evidence to answer a question and reads out Yes/No logits.Class-surface variants are combined with logsumexp, then mapped through a sigmoid.Source: A. Answerability Probe · p. 3
  • On baseline rollouts, mean probed answerability increases during successful searches, while failed searches decline early and remain low; this supports using the probe as a source of action-value targets.Source: A. Answerability Probe · p. 3

B. Privileged Lookahead Teacher

  • The privileged teacher uses 3DGS training scenes to render candidate future views and construct one-step and two-step targets for every candidate motion.The one-step target is the answerability of the view reached after the candidate motion.Source: B. Privileged Lookahead Teacher · p. 3
  • Two-step lookahead expands only the top three first motions ranked by one-step value, and excludes the inverse motion from the next-step choices.Unexpanded candidates use their one-step value as the two-step target; training trajectories are generated offline using two-step values.Source: B. Privileged Lookahead Teacher · pp. 3–4

C. Two-Stage Value Distillation

The student predicts privileged teacher action values using only deployment-available inputs, with training proceeding from one-step to two-step targets.

C. Two-Stage Value Distillation

  • The student predicts teacher lookahead values from the question, recent RGB observations, and candidate-motion text, rather than rendered future views.Its action value is read from Yes/No logits and trained to match teacher values at states visited by the teacher.Source: C. Two-Stage Value Distillation · p. 4
  • The offline dataset records questions, observation histories, candidate motions, and one- and two-step teacher targets, which are fit with soft-label binary cross-entropy.The targets are collected at visited training states.Source: C. Two-Stage Value Distillation · p. 4
  • Training first minimizes the loss on one-step targets, then continues from those parameters on two-step targets.Both stages use the offline teacher dataset.Source: C. Two-Stage Value Distillation · p. 4

D. Inference-Time Policy

At deployment, RFL scores camera motions from learned action values, acts on one selected motion, and updates its observation history without rendering future views.

D. Inference-Time Policy

  • The policy considers motions not yet observed to collide and selects a motion using the learned action values.A collision leaves the pose unchanged while consuming a step, so that motion is excluded until the camera pose changes.Source: D. Inference-Time Policy · p. 4
  • After executing the selected motion, the agent updates its observation history and recomputes action values at the next step.Collision handling uses observed outcomes and no privileged scene geometry.Source: D. Inference-Time Policy · p. 4
  • Candidate scoring requires neither rendering nor generation of future views.The learned LoRA adapters are enabled only for action-value prediction; stopping uses the base VLM with adapters disabled.Source: D. Inference-Time Policy · p. 4

V. EXPERIMENTS

The experiments test viewpoint selection on E3VS-Bench, whose questions require camera motions to reveal viewpoint-dependent evidence, using a scene-disjoint split with 21 unseen test scenes. Methods share episodes, action budget, final-view VQA model, and judge, enabling comparisons across direct control, imitation, and future-view generation.

A. Experimental Setup

  • The experiments ask whether answerability-ranked motions improve viewpoint selection, whether answerability is reliable supervision, which choices affect gains, and whether simulation-trained control transfers to a physical robot.Training scenes permit arbitrary viewpoint rendering for the privileged teacher, while test scenes are unseen.Source: A. Experimental Setup · p. 4
  • The official scene-disjoint split contains 1,406 training episodes, 231 validation episodes, and 377 test episodes across 68, 10, and 21 scenes, respectively.The dataset contains 2,014 episodes across six question types in 99 indoor 3DGS scenes.Source: A. Experimental Setup · p. 4
  • The primary metric is GPT-5.5’s 1–5 mean judge score on answers produced by GPT-5.1 from the final view; success means a score of 5.The study also reports episode length, collision rate, and repetition rate.Source: A. Experimental Setup · p. 4
  • Baselines use the same episodes, action budget, final-view VQA model, and judge, comparing direct camera control, teacher-action cloning, and generated future-view scoring.Direct-control agents include GPT-5.1, Gemini 3.0 Flash, and Qwen3.6-27B; imitation uses Qwen3.6-27B and π0.5, while Cosmos3 (Planning) generates candidate views.Source: A. Experimental Setup · p. 4

B. Main Results

On the E3VS-Bench test set, RFL outperforms direct action prediction with the same base VLM and the strongest evaluated proprietary baseline. Its answerability rises toward the level at human-annotated goal views during test rollouts.

B. Main Results

  • 3.05 mean judge score for RFL exceeds 2.14 for direct action prediction with the same base VLM, 2.36 for Cosmos3 (Planning), and 2.81 for GPT-5.1.Table I reports the main E3VS-Bench test-set results.Source: B. Main Results · p. 5
  • RFL leads on four of six question types, while GPT-5.1 leads OA (2.96 vs. 2.89) and Gemini 3.0 Flash leads OST (3.09 vs. 2.55)..Source: B. Main Results · p. 5
  • RFL’s answerability rises from the first steps to the mean goal-view level by episode end, while GPT-5.1 and Gemini remain below it and Cosmos3 stays near its initial level.Figure 4 plots answerability along test rollouts, with ±1 SE bands and dashed mean answerability at goal views.Source: B. Main Results · p. 5

C. Answerability as a Supervision Signal

Answerability tracks search success and is used as supervision for candidate-wise action values. Ablations examine output targets, lookahead depth, observation history, and the trajectories used to train RFL.

C. Answerability as a Supervision Signal

  • Answerability distinguishes successful from failed validation searches with 0.824 AUROC, versus 0.719 for privileged target-object IoU and 0.743 for episode length.Mean answerability is 0.334 at initial views and 0.685 at ground-truth goal views; it rises toward the goal-view level in successful episodes and remains low in failures.Source: C. Answerability as a Supervision Signal · p. 5

D. Ablation Studies

  • Fine-tuned value prediction scores 2.71 versus 2.47 for action cloning, with 17.2 versus 24.9 steps and 0.62 versus 0.87 repetition.Without fine-tuning, answerability scoring scores 2.04 versus 2.14 for direct action prediction; the comparison uses the same backbone and teacher trajectories.Source: D. Ablation Studies · pp. 5–6
  • Two-step RFL scores 3.05 versus 2.98 for one-step RFL, while the two-step rendering oracle scores 3.19 versus 2.98 for the one-step oracle.At an equal three-epoch budget, two epochs on L1 and one on L2 score 3.05, versus 2.97 for three epochs on L1 alone; the result is consistent with a benefit from two-step targets.Source: D. Ablation Studies · p. 6, Table III · p. 6, Figure 5 · p. 6
  • On validation, action history raises mean judge score from 2.75 to 2.96, and visual history raises it to 3.10.The variants use the same two-stage recipe and differ in whether they receive the current view, recent executed motions, or recent observations.Source: D. Ablation Studies · p. 6, Figure 6 · p. 6
  • Lookahead-teacher trajectories score 3.05 against 2.80 for shortest-path trajectories under the same two-stage recipe.The shortest paths are planned with A* to the goal viewpoint; Figure 7 compares the trajectory choices on the test split.Source: D. Ablation Studies · p. 6, Figure 7 · p. 6

E. Qualitative Results

Qualitative episodes show RFL reaching informative views, but also expose a stopping failure. In physical camera-control trials, RFL has the highest mean judge score overall across the three questions, with one question where GPT-5.1 scores higher.

E. Qualitative Results

  • RFL reaches views revealing queried information and answers two illustrated questions correctly, but in another episode it passes a readable text view and ends where the text is no longer readable.Reanswering from the visited view with highest probed answerability raises the score from 3.05 to 3.16, narrowing the gap to the two-step rendering oracle from 0.14 to 0.03.Source: E. Qualitative Results · p. 6, Figure 8 · p. 7

F. Real-World Experiments

  • Across 90 physical trials, RFL has the highest mean judge score over the three questions, scoring 3.80 on the pot and 3.00 on the oven question.The direct base VLM scores 1.40 on all three; GPT-5.1 scores 3.40 on pot and 1.80 on oven, but leads on the blue-package question, 2.60 versus 2.20 for RFL.Source: F. Real-World Experiments · p. 6
  • The arm’s embodiment limits execution: RFL removes infeasible motions before selection, while the direct base VLM continues generating them after exclusion from its action list.The experiments provide initial evidence of transfer from simulation to physical camera control without test-time rendering.Source: F. Real-World Experiments · pp. 6–7

G. Discussion and Limitations

RFL combines privileged rendered lookahead with distillation to select viewpoints without rendering future views at deployment. The discussion identifies training-time rendering, fine-grained evidence, stopping, and physical execution as limitations.

G. Discussion and Limitations

  • RFL requires counterfactual rendering during training, remains weakest on fine-grained OST evidence, and can fail to stop at a view that exposes the answer.In physical experiments, some simulation-feasible motions were not executable by the arm.Source: G. Discussion and Limitations · p. 7

VI. CONCLUSION

  • A privileged teacher renders candidate actions in 3DGS scenes and scores them by two-step answerability, which the same VLM learns through two-stage soft-label distillation.At deployment, the policy selects viewpoints with a single text probe per candidate, without rendering or generating future views.Source: §VI · p. 7
  • On E3VS-Bench, RFL raised the mean judge score from 2.14 to 3.05; on a 6-DoF arm, it achieved the highest mean score across three questions.It also outperformed action cloning trained on the same supervision.Source: §VI · p. 7
  • The conclusion identifies calibrated stopping and finer-grained or continuous action spaces for physical platforms as future work.Source: §VI · p. 7
Loading 2610.11039v1…