PaperlayerView exact revision on arXiv

Source-linked AI summary

Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Liyang Li, Muzhi Zhu, Zhiyue Zhao, Hengyu Zhao, Ke Liu, Linhao Zhong, Hao Chen, Chunhua Shen

arXiv:2606.01247v1cs.CV

TL;DR

Foundation models are usually evaluated on passive spatial understanding rather than actively moving to reproduce a target viewpoint. This paper introduces TVRBench and a unified post-training framework, achieving 51.4% overall success with visual-action SFT plus Multi-turn GRPO on a 9B model.

  • Problem

    Existing spatial-intelligence benchmarks mostly provide observations in advance, leaving active target-viewpoint reproduction through embodied movement insufficiently evaluated.

  • Method

    The paper introduces TVR and TVRBench, then evaluates expert-trajectory SFT, CoT-SFT, Single-turn GRPO, and Multi-turn GRPO for closed-loop viewpoint control.

  • Results

    51.4% overall success is achieved by visual-action SFT plus Multi-turn GRPO on a 9B model, while CoT supervision and Single-turn GRPO reduce closed-loop performance.

  • Takeaways & Limitations

    TVRBench provides a testbed for measuring and training foundation models that actively perceive and act in 3D environments.

  • Takeaways & Limitations

    Because TVRBench uses simulated discrete poses and one 9B backbone, the findings may not generalize to physical-world control or other model families and scales.

Abstract

from arXiv · show

Humans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.

1 Introduction

The paper introduces Target Viewpoint Reproduction (TVR) and TVRBench to evaluate whether foundation models can actively reproduce a target viewpoint through closed-loop perception and embodied action. TVRBench shows current models remain far from human performance, while visual-action SFT and Multi-turn GRPO substantially improve closed-loop success.

  • Task and benchmark: TVR requires an agent to compare a target image with its egocentric view, infer the viewpoint gap, act through translation, rotation, and head motion, and stop when matched.The task updates spatial beliefs from new observations in a closed perception–action loop.
  • Task and benchmark: TVRBench evaluates active target-viewpoint reproduction in indoor 3D simulations, with protocols diagnosing exploration efficiency, spatial memory, and perception-to-action mapping.The benchmark is designed around explicit viewpoint control rather than passive visual understanding.
  • Benchmark findings: 7.8% success for the strongest open-source model and 12.0% for the strongest closed-source model show TVR remains far from solved, versus 93% human performance on a 100-task subset.The benchmark spans evaluated open- and closed-source MLLMs.
  • Benchmark findings: Models exhibit two bottlenecks: exploiting multi-turn visual history and mapping spatial discrepancies to body translation rather than in-place rotation.Performance drops when viewpoint reproduction requires translation, indicating difficulty beyond static visual recognition.
  • Post-training framework: 50.8% success from visual-action SFT and 51.4% overall after Multi-turn GRPO demonstrate the main post-training gains, while CoT supervision and Single-turn GRPO degrade closed-loop performance.Multi-turn GRPO mainly refines multi-room tasks where SFT is weakest.

2 Related Work

Prior spatial-intelligence work progressed from static and multi-view understanding toward embodied reasoning, but typically used pre-collected observations rather than exploration. TVR instead evaluates explicit target-viewpoint reproduction with coordinated body movement and head rotation, while existing post-training paradigms largely target static QA or per-step control.

  • Spatial Intelligence: Early spatial-intelligence models answered spatial questions from text-image pairs or single observations, later extending to multi-view matching, relation inference, and scene-structure understanding.Embodied work additionally studied embodied question answering and affordance prediction.
  • Spatial Intelligence: Across prior settings, observations were typically pre-collected, asking what is where rather than where the agent should look next.
  • Active Embodied Tasks: Visual navigation specified goals by object class, goal image, or language, but success measured reaching a target region or fulfilling instructions rather than reproducing a target viewpoint.Even ImageNav uses a goal image without making final-observation reproduction the success criterion.
  • Active Embodied Tasks: TVR defines success as explicit viewpoint matching and permits both body movement and head rotation, unlike settings using position reaching, object identification, instructions, teleportation, fixed positions, or one action modality.This combination demands coordination across body and head actions.
  • Post-Training: Static spatial post-training uses SFT, GRPO, or SFT-then-GRPO, while VLA and on-policy RL methods address robot control or navigation rather than closed-loop active perception.In TVR-related experiments, Single-turn GRPO regressed below SFT initialization, whereas trajectory-level Multi-turn GRPO refined supervised priors.

3 Target Viewpoint Reproduction and TVRBench

TVRBench evaluates whether an agent can actively reproduce a target viewpoint in indoor 3D environments using only first-person images and discrete movements. Its benchmark design varies scene scale and target-view visual richness while enforcing exact pose matching.

  • Target Viewpoint Reproduction: In TVR, an agent receives a target image and repeatedly selects actions from first-person observations until stopping or reaching the step limit.Success requires the final pose to exactly match the target image’s viewpoint.
  • Target Viewpoint Reproduction: Nine discrete actions translate the body by 0.25 m, rotate it by 45°, shift camera horizon by 30°, or stop the episode.The agent has no privileged access to pose, target pose, or a scene map.
  • TVRBench Design: TVRBench crosses scene scale with target-view visual richness into four equal-sized categories: Single-easy, Single-hard, Multi-easy, and Multi-hard.Multi-room tasks require traversing rooms, while sparse target views provide fewer visual anchors than object-rich views.
  • TVRBench Construction: 500 evaluation tasks span 125 tasks per category, with shortest paths of 2–8 action steps in single-room scenes and 10–20 in multi-room scenes.Single-room tasks use 120 iTHOR scenes, and multi-room tasks use ProcTHOR-10k two- or three-room homes.
  • Evaluation and Memory: Two memory representations retain trajectory information: action-only memory summarizes prior actions, whereas visual-action memory preserves the full observation-action sequence.The benchmark also reports success, final pose errors |∆p|, |∆θ|, |∆ϕ|, stop rate, false-stop rate, and mean steps.

4 Can Foundation Models Reproduce Target Viewpoints?

TVR remains difficult for foundation models: the best configuration reaches 12.0% overall success, far below humans’ 93.0%. Failures consistently reflect unreliable body translation, excessive revisiting and rotation, and ineffective termination or chain-of-thought reasoning.

  • Main results: 12.0% overall success is the best model result, while humans achieve 93.0% on a balanced 100-task subset and no model exceeds 13%.Dense Qwen3.5 improves from 2.8% at 9B to 7.8% at 27B, but the best closed-source models remain at the 12% ceiling.
  • Main results: F-stop exceeds 75% for 11 of 15 configurations, showing that models usually terminate at the wrong pose.Qwen3.5-9B (VA) reaches 100% F-stop, while GPT-5 is an exception with 0% AO and 27.3% VA.
  • Controlled ablation: 80.5% success follows removal of body-translation actions, whereas allowing only body translation yields 10.0%, identifying translation as a dominant bottleneck.These results come from two 200-task single-room ablations under restricted action spaces.
  • Failure behavior patterns: 3.5 distinct grid positions and 83% pose revisitation characterize Qwen3.5-9B’s inefficient exploration, producing circular walking and looping looks.The model chooses 34.3 actions per episode despite visiting few distinct positions.
  • Failure behavior patterns: 50.8% of actions are rotations, 26.1% are body translations, and 0.1% are Stop, indicating over-rotation with little forward movement or termination.These figures cover 17,159 actions across the benchmark.
  • Failure analysis: ∼10× more response tokens from native thinking mode leave success unchanged, so chain-of-thought does not resolve the observed behavior.Across scales, strategies, and memory formats, the consistent bottleneck is mapping spatial discrepancies to reliable body translation rather than visual matching.

5 Can Post-Training Improve Active Viewpoint Control?

Post-training substantially improves active viewpoint control primarily through visual-action SFT, which reaches 50.8% overall success. Multi-turn GRPO adds targeted multi-room gains, whereas CoT supervision and Single-turn GRPO degrade performance.

  • RL comparison: The results indicate that active viewpoint control benefits from trajectory-level optimization aligned with closed-loop, multi-step structure rather than per-step action matching.Single-turn GRPO regresses, whereas Multi-turn GRPO provides targeted refinement on long-distance multi-room settings.
  • SFT: 50.8% overall success is achieved by VA-SFT without CoT, with strongest performance on single-room tasks and weaker multi-room performance.Single-easy reaches 82.4% and Single-hard 68.8%, while Multi-easy reaches 27.2% and Multi-hard 24.8%.
  • SFT: Visual-action memory improves SFT, while adding CoT reduces success under both action-only and visual-action memory.Without CoT, visual-action memory raises overall success from 44.2% to 50.8%; with CoT, it raises success from 24.8% to 35.6%.
  • Multi-turn GRPO: Multi-turn GRPO raises Multi-easy from 27.2 to 34.4 (+7.2 pp) and Multi-hard from 24.8 to 25.6 (+0.8 pp), while aggregate improvement over VA-SFT is +0.6 pp.Its clearest gains occur on long-distance multi-room splits, while single-room performance remains close to the SFT checkpoint.
  • Single-turn GRPO: Single-turn GRPO reduces overall success by 18.0 pp from AO-SFT, changing 44.2 →26.2, and also degrades stop calibration.Starting from AO-SFT, F-stop rises from 7.9% to 20.6% after Single-turn GRPO.

6 Conclusion

The paper introduces Target Viewpoint Reproduction as a closed-loop embodied task and TVRBench as a benchmark spanning scene scale and target-view visual richness. TVRBench reveals a substantial model–human performance gap, driven mainly by failures to map viewpoint discrepancies to reliable body movement.

  • Task and benchmark: TVR is a closed-loop task requiring target-image reproduction through embodied movement or reorientation.The task combines visual matching with active viewpoint adjustment.
  • Task and benchmark: TVRBench spans scene scale and target-view visual richness.The benchmark evaluates target-view reproduction across varied environmental and visual conditions.
  • Performance gap: 12.0%/7.8% success versus 93.0% humans exposes a large model–human gap on TVRBench.The figures refer to the best closed/open-source models and human performance, respectively.
  • Performance gap: Failures mainly arise from mapping viewpoint discrepancies to reliable body movement.This identifies embodied movement control as the primary failure source highlighted in the conclusion.
  • Post-training: The authors build a unified TVR post-training framework.The supplied passage introduces this framework but does not provide further details in the excerpt.

Limitations

The benchmark’s findings are limited to simulated, discrete-pose viewpoint control with exact-pose success, and post-training conclusions are based on a single 9B open-source backbone.

  • Simulation and task design: TVRBench evaluates simulated environments with a discrete pose grid and exact-pose success, so results do not establish performance for continuous, tolerance-based physical-world control.The environments are AI2-THOR and ProcTHOR-10k; these design choices make difficulty controllable and success unambiguous.
  • Model generality: Post-training conclusions rely on a single 9B open-source backbone, leaving their generality across model families unestablished.The passage explicitly states that broader applicability across model families has not been established.

Ethical Considerations … B.4 Diagnostic metrics

The paper documents TVRBench’s ethical safeguards, appendix extensions, benchmark construction, action space, and diagnostic metrics. The benchmark uses held-out indoor scenes, structured viewpoint tasks, nine discrete actions, and measures both success and failure modes.

  • Ethical Considerations: TVRBench uses simulators and API-accessed models under stated terms, collected human data from five volunteers, and will release code, data, and checkpoints permissively.Evaluation is entirely simulated, though active viewpoint control could enable intrusive real-world uses.
  • A Appendix Overview: The appendices cover benchmark construction, SFT data, post-training configurations, additional quantitative results, and qualitative trajectories.These include four diagnostic metrics, action-only versus visual-action memory, GRPO settings, matching-versus-success comparisons, and representative failures and successes.
  • B.1 Scene splits: TVRBench contains 240 indoor scenes split into SFT, evaluation, and RL pools at a 1:2:3 ratio across scene families.The benchmark balances 120 AI2-THOR single-room scenes with 120 ProcTHOR-10k multi-room scenes.
  • B.1 Scene splits: No scene is shared across pools, so evaluation uses environments unseen during both SFT and RL training.This design makes reported performance measure generalisation rather than scene memorisation.
  • B.2 Task generation: Each task is a start–target pose pair characterized by shortest-path length and the number of visible non-structural objects at the target viewpoint.These dimensions independently represent navigation extent and target-view visual richness.
  • B.3 Action space: Agents choose among nine discrete actions: four 0.25 m translations, two ±45° body rotations, two ±30° head pitches, and Stop.Collisions leave the pose unchanged but consume a step, while exhausting the budget counts as failure.
  • B.4 Diagnostic metrics: Episode success requires issuing Stop at a target-matching pose; otherwise episodes end unsuccessfully when the step budget is exhausted.The formal success indicator is Si = stopi mi, where mi denotes pose matching.
  • B.4 Diagnostic metrics: Diagnostic reporting includes success, stop, pose-match, action-count, final pose-error, per-category rates, and false-stop rate.Per-category rates use N = 125 episodes per difficulty tier, while F-stop measures incorrect stopping among episodes that stop and should be interpreted with Stop rate.

B.5 Human evaluation protocol

The human evaluation established a directly comparable reference by having five volunteers complete balanced evaluation subsets through the same interface, action space, budgets, and success criterion as the models.

  • Evaluation design: Five volunteers each completed a balanced 100-task evaluation subset, with 25 tasks from each of four categories.The categories equally represented scene scale and visual richness.
  • Evaluation design: Participants used a web interface showing current first-person and target images side by side while issuing the same nine discrete actions as the models.The interface supported single-user navigation through the indoor simulator.
  • Evaluation design: 30 actions for single-room iTHOR tasks and 40 for multi-room ProcTHOR tasks were used as the per-task step budgets for both humans and models.This preserved identical evaluation constraints across the two groups.
  • Evaluation design: |∆p| ≤0.01 m, |∆θ| ≤1◦, |∆φ| ≤1◦ defined the pose-matching success criterion for both human and model evaluation.Using the same criterion made the human and model rows directly comparable.

C SFT Data Pipeline … C.3 Two memory formats

The SFT data pipeline generates action-optimal expert demonstrations, optionally annotates them with constrained visual rationales, and represents trajectories using either short action-only or full visual-action memory. These components respectively define the supervision source, rationale format, and visual-history exposure used for SFT.

  • C.1 Rule-based trajectory generation: The oracle planner uses privileged simulator state to generate expert trajectories unavailable to learned models.Inputs include the exact pose, reachable-position graph, and target pose.
  • C.1 Rule-based trajectory generation: Each trajectory follows three phases: view alignment, navigation, and termination with Stop.View alignment adjusts body rotation and head pitch before navigation.
  • C.1 Rule-based trajectory generation: Navigation uses Dijkstra’s shortest path over (x, z, θ), with six unit-cost body-motion actions and a required final heading θ⋆.The planner may rotate away from θ⋆ during navigation but must end at θ⋆.
  • C.1 Rule-based trajectory generation: The deterministic planner produces exactly 1,600 minimum-action-count SFT trajectories, while learned policies use only first-person observations.The demonstrations are action-optimal by construction, and privileged planner state is withheld from learned policies.
  • CoT Annotation Prompt: The CoT annotation prompt requires justification of the ground-truth action rather than proposing an alternative.The rationale must logically support the expert action label even when another action might seem preferable.
  • C.2 CoT annotation with MiMo-V2.5: AO-CoT-SFT and VA-CoT-SFT augment planner triples with short, observation-grounded rationales consistent with optimal action labels.MiMo-V2.5 is prompted to reference visible landmarks, identify heading, distance, or position misalignment, and explain how the action reduces it.
  • C.3 Two memory formats: Action-only memory creates independent single-turn samples, whereas visual-action memory packs each trajectory into one multi-turn sample containing accumulated visual history.Action-only keeps sequences short; visual-action exposes every past observation at each step.

D Post-Training Configuration … E Additional Quantitative Results

The post-training setup combines supervised fine-tuning with single-turn and multi-turn GRPO, differing in context structure, simulator interaction, rewards, and compute. The supplied passages also specify implementation details but contain no substantive additional quantitative results for Section E.

  • D Post-Training Configuration: 4 H100 GPUs for roughly 6 hours are used for one supervised fine-tuning run, compared with 8 H200 GPUs for roughly 10 hours for Multi-turn GRPO and 4 hours for Single-turn GRPO.These are the reported compute budgets for the three post-training procedures.
  • D.1 Supervised fine-tuning: All four SFT variants fully update Qwen3.5-9B while freezing the vision encoder, using AdamW, bf16 precision, a 1 × 10−5 learning rate, cosine decay, and 10% linear warmup.Training uses an image-resolution cap of 262 144 pixels, effective batch size 32, ZeRO-2, and gradient checkpointing; AO variants train for 3 epochs and VA variants for 5.
  • Action-Only Memory Sample (per-step): Action-only memory converts each trajectory step into an independent single-turn sample retaining textual recent-action history but no past observations.The 1,600 SFT trajectories expand to approximately 20,700 per-step samples.
  • SYSTEM: Visual-action memory packs each trajectory into one multi-turn sample, preserving all past observations at every step and yielding exactly 1,600 samples.Trajectories range from a few steps to roughly 30–40 steps, and CoT variants alone include the <think>. . . </think> prefix.
  • D.3 Multi-turn GRPO: Multi-turn GRPO performs on-policy episode optimization in the live TVRBench simulator, initializing from VA-SFT and interleaving model actions with simulator observations.It uses a dedicated 4,800-task RL split and a custom agent loop.

E.1 KL ablation for Single-turn GRPO … F Qualitative Examples

The ablations show that loosening Single-turn GRPO’s KL constraint worsens both success and stopping calibration, while Multi-turn GRPO can learn from scratch and outperform per-step gains in closed-loop evaluation. Qualitative traces link these differences to action-history limitations: untrained policies repeat rotations or walking loops, whereas the trained policy uses visual history to navigate and stop correctly.

  • E.1 KL ablation for Single-turn GRPO: −9.8 pp at β = 0.05 and −15.4 pp at β = 0.01: Single-turn GRPO regresses below AO-CoT-SFT as the KL constraint loosens.F-stop rises from 2.4% at the SFT initialization to 10.9% and 23.5%, respectively.
  • E.2 RL from a base initialisation: 2.8 → 3.6, +0.8 pp: Single-turn GRPO only marginally improves the AO baseline without SFT warm-up.Multi-turn GRPO instead lifts the VA baseline from 0% to 26.2% overall with F-stop = 0%.
  • E.2 RL from a base initialisation: Trajectory-level on-policy RL produces a workable policy from scratch, whereas per-step RL does not.The shaped progress reward provides a learning signal even to a near-random initial policy.
  • E.3 Per-step versus closed-loop accuracy: 72.1% per-step action-matching accuracy and 99.98% format validity coexist with only 9.4% closed-loop benchmark success for the β = 0.01 checkpoint.Its average per-step reward is 0.749, showing that local action matching does not ensure successful closed-loop reproduction.
  • E.3 Per-step versus closed-loop accuracy: 0.78: the β = 0.001 run achieves comparable per-step accuracy, yet the analysis still exposes a gap between per-step and closed-loop performance.The β = 0.01 checkpoint’s closed-loop result is 9.4%.
  • F Qualitative Examples: Untrained Qwen3.5-9B repeatedly rotates in place, issuing 35 consecutive Rotate actions after two early advances until the 40-step budget expires.Its action-only memory cannot indicate that previously tried yaws were rejected, causing the same micro-decision to repeat.
  • F Qualitative Examples: VA-SFT + Multi-turn GRPO aligns with single-room and multi-room targets by translating, rotating, traversing rooms, and stopping at the correct pose.Visual-action memory conditions each step on actual observation history, preventing previously tried yaws from being revisited.
  • F Qualitative Examples: 12 Move actions among only four distinct positions: the untrained agent repeats a short walking loop without closing the target gap.Its action history cannot register that the cells were already visited.
Loading 2606.01247v1…