PaperlayerView exact revision on arXiv

Source-linked AI summary

VCN-Bench: A Video-Contextualized Navigation Benchmark for Spatial Reasoning over Prior Visual Experience

Siqi Zhang, Meng Wei, Chenyang Wan, Shaohao Zhu, Shufan Shen, Xihui Liu, Zhihua Wei, Tai Wang, Jiangmiao Pang

arXiv:2609.34687v1cs.AI

TL;DR

Existing spatial-reasoning benchmarks often stop at offline predictions, and navigation benchmarks entangle spatial reasoning with instruction following or exploration. VCN-Bench uses prior-video context to evaluate closed-loop navigation and diagnose destination identification separately; experiments report limited navigation performance and failures even after correct identification.

  • Problem

    Prior spatial-reasoning benchmarks typically end at offline predictions, while navigation settings combine spatial reasoning with instruction following or exploration, leaving its role in subsequent interaction unclear.

    Source: §1 · pp. 1–2

  • Method

    VCN-Bench evaluates navigation from a prior video covering the initial location and destination, with diagnostic goal identification to distinguish destination-resolution errors from navigation failures.

    Source: §1 · p. 2

  • Results

    Experiments show limited navigation success and a substantial destination-resolution-to-navigation gap, with many episodes failing during navigation despite correct destination identification.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    The benchmark and MV-DualVLN analyses expose challenges in both destination resolution and closed-loop navigation over prior visual experience.

    Source: §1 · p. 2, §7 · p. 11

    VCN-Bench uses static Matterport3D scans and assumes scenes remain unchanged; detecting outdated context and revising decisions after environmental changes is outside its scope.

    Source: E Limitations and Future Work · pp. 26–27

Abstract

from arXiv · show

Spatial reasoning is fundamental to embodied agents, yet it remains unclear whether spatial understanding can be carried forward to guide sequential interactions. Existing spatial-reasoning benchmarks typically terminate at offline predictions, while navigation benchmarks evaluate spatial reasoning as part of instruction following and exploration. We introduce VCN-Bench, a \textbf{V}ideo-\textbf{C}ontextualized \textbf{N}avigation benchmark for probing closed-loop spatial reasoning over prior visual experience in MLLMs. Given a prior video covering both the initial location and destination, the agent is tasked with reasoning out the instruction-specified target and navigating toward it with the inferred spatial context. Built on Matterport3D, VCN-Bench contains five instruction types, 100k training episodes, and 1,250 evaluation episodes. Navigation serves as the primary evaluation, while diagnostic goal identification helps distinguish destination-resolution errors from subsequent navigation failures. We further propose MV-DualVLN, a planning-oriented baseline that jointly leverages prior video and in-episode observations. Experiments reveal limited navigation performance, a substantial destination-resolution-to-navigation gap, and frequent navigation failures even after correct destination identification.

1 Introduction

VCN-Bench addresses the gap between offline spatial prediction and navigation settings where spatial reasoning is entangled with instruction following or exploration. It tests whether spatial context from a prior video can guide destination resolution and closed-loop navigation, with diagnostics separating those stages.

1 Introduction

  • VCN-Bench uses a prior video covering the initial position and target destination to test whether inferred spatial knowledge guides destination resolution and subsequent navigation.Its video-grounded instruction makes the inferred spatial context relevant to both target selection and closed-loop decisions.Source: §1 · p. 2
  • Navigation is the primary evaluation, while diagnostic goal identification separates destination-resolution errors from subsequent navigation failures.The diagnostic asks the model to identify a target room or object in the prior video without navigating.Source: §1 · p. 2
  • VCN-Bench contains five instruction types grouped into Object Goal and Room-to-Object Goal families.Room-to-Object instructions require identifying a target room through Count, Order, Distance, or Orientation relations before specifying an object.Source: §1 · p. 2
  • Built on Matterport3D, VCN-Bench has 100k training episodes and 1,250 evaluation episodes, and experiments find limited navigation success and many failures after correct destination identification.MV-DualVLN is introduced as a baseline that jointly reasons over the prior video and its own observations to plan pixel-space actions.Source: §1 · p. 2

2 Related Work

Prior spatial-reasoning benchmarks measure offline prediction, while navigation benchmarks evaluate spatial reasoning alongside instruction following or exploration. VCN-Bench is positioned to probe reasoning over prior visual experience in closed-loop navigation.

Spatial Reasoning Evaluation for MLLMs

  • Spatial-reasoning benchmarks use offline VQA or grounding over fixed inputs, whereas VCN-Bench targets closed-loop reasoning through video-contextualized navigation.The cited benchmark families include 3D VQA and multi-image spatial reasoning [Zhang et al., 2025b; Yang et al., 2025a; Lin et al., 2025].Source: §2 · p. 3

Navigation Benchmarks

  • VLN benchmarks emphasize instruction fidelity and landmark recognition, while goal-oriented navigation emphasizes fast exploration via commonsense knowledge.These task settings evaluate spatial reasoning jointly with instruction following or exploration.Source: §2 · p. 3
  • Existing navigation evaluations can confound spatial reasoning with model-dependent exploration and trajectory coverage, while VCN-Bench supplies scene information as pre-recorded video.This centers evaluation on reasoning over prior visual experience and using spatial context to guide navigation.Source: §2 · p. 3

3 VCN-Bench

VCN-Bench pairs prior videos with instructions grounded in their visual context and evaluates navigation using five instruction types. Its construction draws on Matterport3D tours and produces a dataset with disjoint training and test scenes.

3.1 Task Formulation

  • Each episode provides an RGB prior video covering both the agent’s initial position and the target, then evaluates action sequences using current RGB and depth observations.Evaluation scenes are disjoint from training scenes, and prior video is the only scene-specific information available before navigation.Source: §3.1 · p. 4

3.2 Instruction Design

  • Five instruction types form Object Goal and Room-to-Object Goal categories, covering instance localization and room selection through count, order, distance, or orientation relations.Instructions are grounded in the prior video, include controlled distractors, and are designed so the target can be determined from that video and the agent’s initial state.Source: §3.2 · p. 4

3.3 Benchmark Construction

  • Construction samples variable-length prior videos from Matterport3D floor tours, grounds templated instructions in them, and filters duplicate or too-close episodes.The pipeline also uses human and automated checks to catch video-generation failures, instruction-semantic changes, and ambiguous evaluation episodes.Source: §3.3 · p. 5

3.4 Dataset Statistics

  • The dataset has 103,055 training episodes from 61 scenes and 1,250 test episodes from 11 disjoint unseen scenes, with 250 test episodes per instruction type.Prior videos average 157.9 frames, and episodes average 122.1 steps—more than twice the 55.8-step average reported for VLN-CE.Source: §3.4 · pp. 5–6

4 Baseline

MV-DualVLN is a planning-oriented baseline designed to retain spatial context from the prior video and navigation history. It identifies a goal in the video and predicts pixel-space navigation goals using multi-view observations.

4 Baseline

  • MV-DualVLN is built on DualVLN System 2 and targets spatial memory retention, reasoning over prior video, and goal-directed navigation planning.DualVLN System 2 plans by predicting turns or mid-term pixel-space waypoints, while System 1 executes low-level trajectories.Source: §4 · p. 6

4.2 Design Details

  • MV-DualVLN uses an oracle executor to reach predicted pixel-space waypoints, isolating navigation planning from low-level control failures.Feature pooling downsamples each image’s 12 × 12 feature map to 3 × 3, reducing tokens per image by 16×; training pixel goals use frontier waypoints and ultimately the target object’s centroid.Source: §4.1 · p. 6, §4.2 · p. 6

4.1 Model Overview

  • The model selects a target frame in the prior video, then predicts a pixel goal and termination signal using spatial context jointly encoded from the video and navigation history.Its inputs include the instruction, a 50-frame prior video, navigation history, and multi-view observations at the initial and current positions.Source: §4.1 · p. 6

4.2 Design Details

  • Navigation history records observations associated with rotations as well as forward actions, retaining visual continuity while adding views from different directions.For a right-view selection, the history includes both front and right views.Source: §4.2 · p. 6

5 Evaluation Protocols and Baselines

The evaluation treats video-contextualized navigation as the primary task and goal identification as a diagnostic for separating destination-resolution errors from navigation failures. It compares hierarchical and end-to-end navigation baselines and evaluates frame selection for instruction-specified goals.

5 Evaluation Protocols and Baselines

  • The protocol evaluates navigation as the main task and uses goal identification to diagnose whether failures arise from destination resolution or closed-loop interaction.Table 1 compares navigation performance; UniNavid directly outputs low-level actions, while other models use an oracle executor for predicted waypoints.Source: §5 · p. 7, Table 1 · p. 8

5.1 Primary Task: Video-Contextualized Navigation

  • Navigation is the primary behavioral probe of whether agents use prior-video information to make sequential decisions toward a destination.Success Rate (SR) is the primary metric; success requires stopping within 1 m of a valid target viewpoint, while Room Success Rate (RSR) accepts ending anywhere in the target room.Source: §5.1 · p. 7

5.2 Navigation Baselines

  • Hierarchical baselines select a target frame before a separate navigation model acts, whereas end-to-end baselines jointly resolve the destination and navigate using video and online observations.The hierarchical pairing uses goal identification followed by MTU3D; end-to-end methods include MV-DualVLN, 3D-Mem, and Uni-Navid.Source: §5.2 · p. 7

5.3 Diagnostic Task: Goal Identification

  • Goal identification tests destination resolution without navigation by asking the model to select the prior-video frame matching the instructed destination.Rid-SR measures selection of a frame from the target room for R2O instructions, while Oid-SR measures whether the target object is visible for Object-goal instructions.Source: §5.3 · p. 7

5.4 Goal Identification Baselines

  • Goal-identification baselines receive a common instruction format and return one frame index, with proprietary models evaluated in thinking and nothinking variants.The evaluated set also includes open-source MLLMs, including the Qwen3-VL family.Source: §5.4 · p. 8

6 Results and Analysis

Across VCN-Bench, navigation remains difficult, and diagnostic results show that both identifying destinations and navigating after identification pose challenges. Analyses find a gap between goal identification and navigation, and show that prior-video context and navigation history matter to performance.

6.1 Navigation Performance

  • 27.6% SR is the best navigation result, achieved by MV-DualVLN, with success in fewer than one-third of episodes.The results describe current models as struggling to convert scene-specific visual experience into effective sequential navigation decisions.Source: §6.1 · p. 8
  • 17.6% SR with Gemini-3-Flash-Thinking-selected frames and 20.2% SR with Oracle frames are reported for hierarchical MTU3D.The authors conjecture that MTU3D lacks prior-video spatial context and relies on CLIP global features, hindering fine-grained discrimination among same-category rooms or objects.Source: §6.1 · pp. 8–9
  • Fine-tuned 3D-Mem reaches 18.1% SR, and MV-DualVLN improves on it by 9.5 percentage points; MV-DualVLN SR is higher on Instance, Order, and Count than Distance and Orientation.The task-wise SR values for MV-DualVLN are 34.4% on Instance, 34% on Order, 30.8% on Count, 20.8% on Distance, and 18% on Orientation.Source: §6.1 · p. 9

6.2 Diagnostic Goal Identification Results

  • Models generally identify Count and Order destinations more successfully than Distance and Orientation destinations, and most find Object goals harder than R2O goals.Thinking variants improve results at the same scale; Gemini-3-Flash-Thinking gains 6.8% average Rid-SR and 7.6% Oid-SR over nothinking, while Qwen3-VL-8B-Thinking gains 16.4% average Rid-SR and 12.0% Oid-SR over Qwen3-VL-4B-Thinking.Source: §6.2 · p. 9
  • Goal identification remains substantially below human performance, while category-based chance retrieval is insufficient to identify destinations.On 100 sampled episodes, humans outperform Gemini-3-Flash-Thinking by 44.6% average Rid-SR and 55.0% Oid-SR, but still find room-distance and orientation comparisons challenging without a top-down map.Source: §6.2 · p. 9, Table 2 · p. 10

6.3 Analyses and Ablations

  • When first-step goal identification is correct, MV-DualVLN achieves 37.8% SR versus 18.6% when it is incorrect, yet 62.2% of correctly identified episodes still fail; using only a target frame instead of the full prior video reduces SPL by 6.9%, SR by 7.9%, and RSR by 9.1%.The authors conclude that MV-DualVLN leverages scene context from the prior video throughout navigation.Source: §6.3 · p. 10
  • Removing in-episode history with 50-frame video decreases SR by 3.5% and SPL by 4.2%, while Qwen3-VL variants show negative correlations with all three prior-video attributes analyzed.Those attributes are frame count, number of rooms covered, and covered area; Gemini-3 shows a negative frame-count correlation only on Order tasks.Source: Table 3 · p. 10, §6.3 · pp. 10–11, Figure 7 · p. 11
  • MV-DualVLN shows a positive correlation between goal identification and navigation, but average R2O Rid-SR of 43.1% exceeds RSR of 31.7%, and Object Oid-SR of 49.6% exceeds SSR of 39.2%.These comparisons yield gaps of 11.4 and 10.4 points, respectively.Source: §6.3 · p. 9, Figure 6 · p. 11

7 Conclusion

VCN-Bench probes closed-loop spatial reasoning from prior video, with video-grounded instructions and diagnostic goal identification. Experiments show limited navigation performance and many failures even after correct destination identification.

7 Conclusion

  • VCN-Bench evaluates closed-loop navigation from prior visual experience and uses goal identification to distinguish destination-resolution errors from later navigation failures.The authors report limited navigation performance and that many episodes still fail after the destination is correctly identified.Source: §7 · p. 11

A More Related Works

  • Existing MLLM navigation models leave closed-loop spatial reasoning underexplored, and the paper evaluates them while proposing MV-DualVLN as a reference baseline.The related-work discussion contrasts zero-shot planners with VLA models fine-tuned on large-scale embodied data.Source: A More Related Works · p. 16

B.1 Tour video VS. Prior video

  • Prior videos are extracted as variable-length sliding windows from floor tours, and duplicate episodes from overlapping windows are removed using instruction tags.The window size changes in increments of 10 frames, and each new window starts in a region different from the previous one.Source: §B.1 · p. 16

B.2 Region Annotation

  • Region annotation uses Matterport3D labels, excludes rooms with ambiguous boundaries from R2O starts and targets, and manually resolves frames assigned to multiple rooms.The multiple-room issue affects 5.8% of frames.Source: §B.2 · p. 16

B.3 Object Visibility Definition

  • An object counts as visible when it is within 3m, over 60% of its front surface passes a depth-consistency test, and it occupies at least 50 × 50 visible pixels.The depth-consistency threshold is 0.1m in a 384 × 384 frame.Source: §B.3 · pp. 16–17

B.4 Instruction Design Principles

  • Instructions cover object-instance identification and room-to-object selection through Count, Order, Distance, or Orientation relations, with controlled visitation conditions to disambiguate targets.For example, the Order instructions constrain which rooms appear before or after the initial room in the video.Source: §B.4 · pp. 17–19

B.5 Instruction Rephrasal

  • Gemini-3.1-Flash rephrases template instructions, while validation preserves task-critical phrases and checks that the rewritten instruction retains the original intent.Distance instructions, for example, must retain that candidate rooms are among those visited in the video.Source: §B.5 · p. 19

B.6 Data Validation Details

The appendix details data-quality checks, benchmark statistics and examples, navigation evaluation, and implementation choices for evaluated baselines and goal identification.

B.6 Data Validation Details

  • Tour videos are manually checked for simulator failures, rephrased instructions undergo two-level semantic validation, and experts verify evaluation episodes are unambiguous and reachable.Failed rephrasings revert to the template-based instruction.Source: §B.6 · pp. 19–20

B.7 Dataset Statistics; B.8 Benchmark Examples

  • Benchmark statistics cover prior-video frame counts, rooms and area, plus instruction length, shortest navigation distance, and action steps; examples show five instruction types and episode locations.Figure 10 presents the statistics, while Figure 9 illustrates example instructions, observations, and scene trajectories.Source: §B.7 · p. 20, Figure 10 · p. 20, §B.8 · p. 20

C.1 Navigation Success Definition

  • Navigation succeeds when the stopping position is within 1m of a viewpoint sampled around the target object, a metric the authors describe as slightly more lenient than traditional object navigation.Viewpoints are sampled at 0.1m intervals within 0.6m around the object bounding box.Source: §C.1 · p. 20

C.2 Agent Setup; C.4 Implementation Details of Goal Identification

  • Evaluation uses a Hello Robot Stretch with forward, left-turn, right-turn, and stop actions; agents receive a room-tour video, current observations, and an instruction.The goal-identification prompt is referenced in Figure 11.Source: §C.2 · pp. 20–21, Figure 11 · p. 21

C.3 Implementation Details of Navigation Baselines; MTU3D

  • The hierarchical paradigm separates goal identification from navigation but limits the navigator’s use of video structure, whereas end-to-end planning can use both prior video and navigation history.MTU3D navigates first toward a selected target frame, then performs object-goal navigation because the frame may not be close enough to the target.Source: §C.3 · p. 21, MTU3D · pp. 21–22

MV-DualVLN

  • MV-DualVLN pools image features into 3 × 3 tokens and is trained for five epochs with its vision encoder frozen.Training uses bfloat16 on 32 A100 GPUs and takes about 22 hours.Source: MV-DualVLN · p. 22

3D-Mem

  • 3D-Mem chooses waypoints from memory and frontier snapshots, using object centroids for localized target snapshots and predefined distances for frontier snapshots.The benchmark evaluation includes zero-shot Qwen3-VL-8B-Instruct and a fine-tuned Qwen3-VL-4B-Instruct setting.Source: 3D-Mem · p. 22

Uni-Navid

  • Uni-Navid is adapted by prepending the prior video to its in-episode navigation history.It is described as a video-based model trained on navigation and video-question-answering data.Source: Uni-Navid · p. 22

C.5 Human Evaluation

Three human evaluators assessed goal identification on the tiny benchmark with unlimited time and access to the prior video, instruction, multi-view observations, and initial position.

Setup

  • Three evaluators could review the prior video repeatedly and select target frames without a time limit; their averaged performance is reported.They also received the instruction, multi-view observations, and episode initial position.Source: §C.5 · p. 23

Human errors

  • Human errors arise from judging distance and direction without proprioceptive feedback, ambiguity in initial-pose localization, and careless frame selection or missed spatial descriptors.Four static egocentric views can leave visually similar rooms difficult to distinguish without active exploration.Source: §C.5 · p. 23

D.1 MLLM Errors Analysis

The analysis categorizes goal-identification errors and finds different dominant error types across instruction families, alongside common room-position errors.

D.1 MLLM Errors Analysis

  • R2O errors include position, disturbance, cognition, and out-of-range errors, while Object Goal errors include instance, perception, and out-of-range errors.Position errors select a frame inside the target room but outside the room; disturbance errors select the wrong room of the correct type.Source: §D.1 · p. 23
  • Gemini-3 has no out-of-range errors, while Qwen3-VL has a non-negligible number, which the authors associate with hallucination issues.The comparison analyzes Gemini-3-flash-thinking and Qwen3-VL-8B-Thinking.Source: §D.1 · pp. 23–24
  • Position errors are common across R2O tasks; the authors attribute this to restricted views inside rooms and difficulty representing position and relations between frames.A more comprehensive view of a room may be available from outside it.Source: §D.1 · p. 24
  • Disturbance errors predominate in Distance and Orientation tasks, while cognition errors predominate in Count and Order tasks.The authors relate these patterns to the challenges associated with each task family.Source: §D.1 · p. 24
  • Perception error is the primary Instance-task error, which the authors suggest may reflect the many relations between target objects and surrounding objects in the instructions.Source: §D.1 · p. 24

D.2 Analysis on Spatial Context Utilization

Controlled interventions test how MV-DualVLN uses instruction relations and prior-video order, while oracle-frame navigation and qualitative examples probe its remaining behavior.

D.2 Analysis on Spatial Context Utilization

  • Using ground-truth target frames increased SPL by 3.7%, SR by 5.5%, and RSR by 6.0%.Only the target-frame prefix changed; other inputs and navigation decisions remained unchanged.Source: §D.2 · p. 25
  • Counterfactual interventions degraded performance across tasks, with SR dropping by 27.6% on Order tasks.The authors interpret this as evidence that MV-DualVLN relies on relational semantics in instructions.Source: §D.2 · pp. 25–26, Table 4 · p. 26
  • Shuffling frames cut SR by 20.0% on Count and 24.0% on Order, versus 9.2% on Distance and 8.8% on Orientation.Instance SR also fell by 17.6%; the authors conjecture that preserved visual content and frame overlap still support geometric reasoning despite lost temporal cues.Source: §D.2 · p. 26, Table 4 · p. 26

D.3 Qualitative Results of MV-DualVLN

  • Qualitative examples show MV-DualVLN selecting the next waypoint via a pixel goal and locating the target object in the final step.The section presents three visualization samples.Source: §D.3 · p. 26, Figure 15 · p. 26

E Limitations and Future Work

VCN-Bench evaluates prior-visual spatial reasoning in static Matterport3D scenes with unchanged environments and videos covering both the start and destination. The authors identify limited model coverage and partial real-world observations as scope boundaries, and outline extensions to task design and visual representations.

E Limitations and Future Work

  • VCN-Bench assumes Matterport3D scenes remain unchanged between the prior-video and navigation phases, leaving agents’ detection of outdated context and decision revision outside scope.Environmental changes can invalidate object-level evidence or alter local traversability; most episodes focus on Room-to-Object goals and relatively stable room-level relations.Source: E Limitations and Future Work · pp. 26–27
  • Model coverage, particularly for proprietary MLLMs, is limited by API costs and heterogeneous visual-input constraints.Some APIs accept fewer frames than the average prior video, and aggressive downsampling may remove destination evidence or disrupt instruction-relevant trajectory structure.Source: E Limitations and Future Work · p. 27
  • Prior videos cover both the navigation start and destination, unlike real-world settings that may offer only partial observations and require efficient exploration.Source: E Limitations and Future Work · p. 27
  • Future work proposes evaluating reasoning over known and unknown regions and exploring textual scene descriptions or structured scene graphs as visual-experience representations.The stated goal for these representations is improved memory compression and spatial understanding.Source: E Limitations and Future Work · pp. 27–28

F Social Impact

  • The authors identify potential benefits for home-service and assistive robots, while warning that Matterport3D’s Western-style indoor scenes may limit performance in other living environments.The stated applications include fetching objects and indoor guidance; the potential bias may affect non-Western layouts, accessible housing, and minority living environments.Source: F Social Impact · p. 28

G License and Access

  • Matterport3D use requires following its original license and obtaining authorization from its hosts; EmbodiedScan annotations require an access request and compliance with their license.Source: G License and Access · p. 28
Loading 2609.34687v1…