PaperlayerView exact revision on arXiv

Source-linked AI summary

HorizonNav: Geometry-Guided Video Diffusion for Anticipatory 3D Robot Navigation

Hajira Amjad, Valerii Serpiva, Viacheslav Liubimov, Didar Seyidov, Dzmitry Tsetserukou

arXiv:2610.13849v1cs.RO

TL;DR

Robots navigating unfamiliar environments must reason about regions beyond the current view, but temporally coherent generated views do not by themselves ensure geometrically aligned depth. HorizonNav uses prompted video generation and landmark-guided depth alignment to build a spatial hypothesis for pre-motion planning. In its horizon study, both planning-feasible scenes and successful RRT trials increase across the reported settings.

  • Problem

    Visual SLAM leaves regions beyond the current field of view unknown, while temporally coherent dense depth does not by itself ensure accurate multi-view geometry.

    Source: §I · p. 1

  • Method

    HorizonNav generates prompted virtual views from one RGB observation, then combines SLAM poses and landmarks with dense depth predictions to align a plausible spatial map for planning.

    Source: §I · p. 1, §VII · p. 7

  • Results

    Planning-feasible scenes increase from 13/19 to 15/19 and successful RRT trials increase from 62/90 to 75/90 across the 19-scene horizon study.

    Source: §VII · p. 7

  • Takeaways & Limitations

    Generated observations can support pre-motion planning in some scenes, but successful reconstruction alone is insufficient when obstacle boundaries or free-space connectivity are not preserved.

    Source: §VI · pp. 6–7

    The study is limited to indoor scenes, one evaluated video generator, fixed structured prompting without a prompt ablation, and no protocol-matched single-image-depth baseline.

    Source: §VI · p. 7

Abstract

from arXiv · show

We present HorizonNav, a video diffusion-based framework for anticipatory robot navigation that transforms a single RGB image and a text prompt into a geometrically aligned representation of plausible surrounding 3D space. Given the initial observation and a textual description of the reconstruction objective, a vision-language model, guided by designed instructions, interprets the scene and constructs structured prompts for a video diffusion model. The resulting video sequence provides plausible future observations of the environment, which are subsequently processed using monocular ORB-SLAM3 for camera pose estimation and Video Depth Anything for dense depth prediction. Our central contribution is a geometry-guided depth alignment stage that combines the estimated camera poses with triangulated visual landmarks to constrain dense depth predictions across multiple views. This alignment improves the geometric consistency of the reconstructed 3D environment, addressing inconsistencies that temporal coherence of the generated video alone cannot resolve. The experimental evaluation of navigation in imagined 3D environment shows that across a 19-scene horizon study, planning-feasible scenes increase from 13/19 at P20 to 15/19 at P81, while successful RRT trials increase from 62/90 to 75/90. In a separate corridor study, 6 of 25 evaluable generated-camera trajectories violate the reconstructed collision constraints; RRT planner recovers feasible alternatives in 5 of those 6 cases. A larger archive contains 58 usable reconstructions from 67 attempted cases, of which 36 support a downstream plan and 22 do not, exposing reconstruction and free-space failure modes rather than hiding them without lidar. Across 15 corridor reconstructions, the mean reconstruction error is approximately 0.30 m at matched structural landmarks.

I. INTRODUCTION

HorizonNav uses generated future views to build a cautious, geometrically aligned spatial hypothesis from one image and a language task. It treats that map—not the generated camera path—as the basis for navigation planning.

I. INTRODUCTION

  • Visual SLAM reconstructs observed structure while regions beyond the current field of view remain unknown, so spatial hypotheses must not be treated as ground truth.Temporally coherent depth alone does not ensure accurate multi-view geometry.Source: §I · p. 1
  • Generated trajectories need not be safe to execute, and deformed views or inconsistent occlusions can alter free-space connectivity.Source: §I · p. 1
  • HorizonNav transforms one RGB image and a language task into plausible surrounding space using VLM-guided exploration, generated views, SLAM geometry, and aligned depth.ORB-SLAM3 supplies camera poses and sparse landmarks; Video Depth Anything supplies dense depth, which is aligned across views.Source: §I · p. 1, Figure 1 · p. 2

A. Generated Futures for Robot Planning

B. Geometric Reconstruction from Generated Observations

HorizonNav asks whether generated observations can support a planning-oriented spatial hypothesis without changing the generator. Its pipeline turns prompted virtual exploration into a fused 3D representation, while treating dense depth and downstream planning as distinct components.

B. Geometric Reconstruction from Generated Observations

C. 3D Reconstruction and Navigation Planning

  • Video Depth Anything's temporal consistency is not assumed to guarantee geometric correctness; it is evaluated separately as a densification diagnostic.Source: C. 3D Reconstruction and Navigation… · p. 2
  • ORB-SLAM3 supplies primary sparse multi-view geometry and RRT is the downstream planner; neither is presented as an algorithmic contribution.The research question is whether generated observations provide enough geometric evidence for pre-motion planning while keeping the plan independent of generated camera motion.Source: C. 3D Reconstruction and Navigation… · p. 2

III. SYSTEM ARCHITECTURE

  • The VLM analyzes visible landmarks, obstacles, traversable regions, and occlusions, then specifies exploration direction, motion primitives, visual anchors, and regions needing observation.Prompt constraints distinguish visible evidence from uncertain areas and discourage unjustified objects or scene changes.Source: §III · p. 3
  • Wan 2.2 conditions candidate first-person videos on the standardized initial frame and exploration command, continuing the observation along exploratory motion.Multiple candidate sequences can be sampled when the unobserved space is ambiguous.Source: §III · p. 3
  • ORB-SLAM3 estimates camera poses and sparse landmarks while Video Depth Anything predicts dense relative-depth maps, which are aligned and fused into a unified point cloud.Source: §III · p. 3

A. Trajectory-Guided Alignment of Relative Depth and Sparse SLAM Landmarks

The alignment stage uses sparse SLAM landmarks and camera poses to calibrate dense relative-depth predictions into the SLAM coordinate frame. It filters unreliable landmark observations before using them as geometric anchors.

A. Trajectory-Guided Alignment of Relative Depth and Sparse SLAM Landmarks

  • ORB-SLAM3 provides monocular camera poses and triangulated landmarks, while Video Depth Anything predicts a dense relative-depth map for each frame.Source: A. Trajectory-Guided Alignment of… · p. 3
  • Because dense predictions may have scale and offset misaligned with SLAM geometry, sparse landmarks anchor depth calibration across views.Source: A. Trajectory-Guided Alignment of… · p. 3
  • Only landmarks with positive depth, valid image projection, low reprojection error, and sufficient observations are retained for alignment.The relative depth at each retained landmark location is obtained by bilinear interpolation.Source: A. Trajectory-Guided Alignment of… · p. 3

1) Robust scale and shift estimation:

The method calibrates relative depth with a per-frame affine scale and shift fitted to sparse SLAM landmark depths. Temporal regularization discourages frame-to-frame parameter fluctuations while allowing gradual change.

1) Robust scale and shift estimation:

2) Local residual correction:

Global scale and shift correct low-frequency depth errors, but local structures may remain misaligned. Landmark residuals are propagated with an edge-aware kernel to preserve depth discontinuities.

2) Local residual correction:

  • Global scale and shift correct low-frequency depth errors but may leave local structures misaligned.Sparse residuals at landmark locations are then computed for local correction.Source: 2) Local residual correction · p. 3
  • An edge-aware kernel propagates landmark residuals to nearby pixels while limiting propagation across object boundaries.The image-dependent term preserves depth discontinuities in the final corrected depth map.Source: 2) Local residual correction · p. 3

3) Dense point cloud reconstruction and fusion:

Corrected dense depths are back-projected and transformed into a common world frame before point-cloud fusion. RRT then plans in a conservative free space derived from the reconstructed geometry, which remains a hypothesis requiring cautious execution.

3) Dense point cloud reconstruction and fusion:

B. RRT-Based Navigation in the Reconstructed 3D Space

  • RRT plans in a conservative free configuration space created by converting the point cloud to an occupancy map and inflating observed obstacles by robot radius and safety margin.Source: B. RRT-Based Navigation in the… · p. 4
  • RRT iteratively samples configurations and extends the tree only along collision-free segments, without requiring a differentiable map or accurate analytical environment model.The method is suited to irregular boundaries, narrow passages, and incomplete geometry.Source: B. RRT-Based Navigation in the… · p. 4
  • The planner prioritizes quickly finding a feasible safe path over a globally shortest one and can be rerun after map updates; obstacle inflation and online perception remain necessary because unseen space is only hypothesized.The framework applies to ground robots and UAVs, with UAV planning accounting for altitude.Source: B. RRT-Based Navigation in the… · p. 4

IV. EXPERIMENTAL EVALUATION

The evaluation tests whether generated virtual views provide geometric evidence for independent pre-motion planning, distinguishing map-level feasibility from physical accuracy. It examines planning across view horizons and whether generated camera motion can be evaluated separately from the executed plan.

IV. EXPERIMENTAL EVALUATION

  • The evaluation separates planning utility, independence of generated camera motion from the executed plan, geometric diagnostics, and physical transfer; reconstructed-map feasibility is not proof of correct unseen geometry.Source: §IV · p. 4

A. Planning Utility Across Virtual-View Horizons

  • The horizon study evaluates P20, P40, P60, and P81 by counting scenes with at least one RRT path and seed-level RRT success, measuring planning utility rather than physical reconstruction accuracy.Source: A. Planning Utility Across… · p. 4

B. Alternative Virtual Exploration and Independent Planning

  • The corridor study tests whether generated camera motion supplies virtual observations without serving as an assumed executable robot trajectory.Source: B. Alternative Virtual Exploration and… · p. 4
  • Two trigger observations, four exploration commands, and two seeds yield 16 videos, evaluated at P40 and P81 for 32 branch-prefix cases.Source: B. Alternative Virtual Exploration and… · p. 4
  • A branch-prefix case is evaluable only when the recovered camera trajectory and independent plan share a metric obstacle representation; trajectory feasibility is checked against the RRT collision geometry.A trajectory is infeasible if any point violates the 0.20 m collision criterion.Source: B. Alternative Virtual Exploration and… · p. 4

C. Geometric Diagnostics

The diagnostics distinguish matched-structure agreement from global scale consistency, while the flight study tests physical execution separately from map-level feasibility. The evaluation uses external LiDAR only as a reference and computes complete waypoint paths before takeoff.

C. Geometric Diagnostics

  • Geometric diagnostics distinguish local agreement at selected features from consistency of recovered global metric scale.Source: C. Geometric Diagnostics · p. 4
  • The S06 thin-aperture comparison tests whether corrected dense VDA+SLAM depth changes traversability relative to sparse SLAM after timestamp and metric-frame alignment.Source: C. Geometric Diagnostics · p. 5
  • Corridor geometry is compared with an independent LiDAR map using fixed-scale rigid registration and separate similarity registration; matched landmarks provide diagnostics, not a complete surface-reconstruction benchmark.LiDAR is reserved for evaluation and is unavailable during reconstruction or planning.Source: C. Geometric Diagnostics · p. 5, Figure 2 · p. 5

D. Real-World Flight Validation

  • Six quadrotor trials test whether precomputed map-feasible paths execute physically, with no in-flight observations used to modify the reconstruction or replan.A flight succeeds when its recorded trajectory crosses the gate plane within the measured aperture; motion capture supplies state feedback and records the trajectory.Source: D. Real-World Flight Validation · p. 5

V. RESULTS

Planning utility generally rises with additional generated viewpoints, but reconstruction success does not guarantee a plan-feasible map. Independent RRT planning can reject generated camera motion and find alternatives.

A. Planning Utility Across Virtual-View Horizons

  • Planning-feasible scenes rise from 13/19 at P20 to 15/19 at P81, and successful RRT trials rise from 62/90 to 75/90.The trend is not monotonic for every individual reconstruction, and these counts measure planning utility rather than surface accuracy.Source: A. Planning Utility Across… · pp. 5–6

B. Reconstruction Yield and Failure Analysis

  • 58/67 attempted cases yield usable reconstructions; 36 support a feasible plan and 22 do not.The resulting 37.9% planning failure rate among usable reconstructions is reported separately from reconstruction yield.Source: B. Reconstruction Yield and Failure… · p. 6
  • Planning failures reflect limited or degenerate camera motion, degraded tracking or reconstruction, and geometry that closes narrow free-space connections.Longer generation can add viewpoints but can also hurt when later imagined geometry expands into the planning corridor.Source: B. Reconstruction Yield and Failure… · p. 6

C. Independent Planning over Generated Futures

  • Among 25 evaluable corridor cases, 6 generated-camera trajectories violate the 0.20 m collision criterion, while RRT finds feasible alternatives in 5 of those 6 cases.Both are checked against the same reconstructed obstacle geometry; generated motion can aid reconstruction without being suitable to execute.Source: C. Independent Planning over Generated… · p. 6
  • P40 feasibility patterns remain stable across 20 planner seeds, rather than depending on one favorable RRT seed.Source: C. Independent Planning over Generated… · p. 6

D. 3D Geometric Diagnostics

External geometric diagnostics assess global scale and agreement at selected landmarks, while a thin-aperture test exposes a traversability failure in corrected dense geometry.

D. 3D Geometric Diagnostics

  • The LiDAR diagnostic registers reconstructions to an independent corridor map and compares selected structural landmarks after rigid alignment with scale fixed.The selected features include door-frame corners, wall intersections, and corners of fixed objects; this is not a complete surface-accuracy measure.Source: D. 3D Geometric Diagnostics · p. 6
  • Allowing an additional global scale parameter changes correspondence RMSE only from 0.325 m to 0.324 m, supporting global metric-scale consistency after anchoring.This does not establish local surface accuracy or correctness of unseen geometry.Source: D. 3D Geometric Diagnostics · p. 6
  • In the S06 thin-aperture diagnostic, corrected dense geometry closes an opening that remains traversable in sparse SLAM.The comparison is a failure mode, not evidence that dense fusion is more accurate than sparse SLAM.Source: D. 3D Geometric Diagnostics · p. 6, Table IV · p. 6

VI. DISCUSSION

HorizonNav treats generated video as virtual sensing for pre-motion planning, but its results support planning-oriented geometric evidence rather than recovery of the true unseen scene. The discussion distinguishes trackability, planning utility, reconstruction quality, and transfer to physical flight.

VII. CONCLUSION

  • The study supports useful pre-motion planning in some scenes, not full 3D scene reconstruction or proof that the unseen scene has been recovered.Dense-depth errors can create false obstacles and reduce planning utility.Source: §VI · p. 6, §VII · p. 7

VI. DISCUSSION

  • The larger archive yields 58 usable reconstructions from 67 attempts, but only 36 of 58 support planning, showing reconstruction alone is insufficient.Obstacle boundaries and free-space connectivity must also be preserved for planning.Source: §VI · p. 7
  • Among 25 evaluable corridor cases, generated-camera motion violates reconstructed collision constraints in six, and independent RRT finds alternatives in five of those six.The generated motion supplies observations; it is not the executed robot trajectory.Source: §VI · p. 7
  • Five of six physical flights pass through the gate using precomputed paths, but this is limited evidence in the tested configuration, not statistical validation of general reconstruction accuracy.The study is limited to indoor scenes, one video generator, fixed structured prompts, and no protocol-matched single-image-depth baseline.Source: §VI · p. 7

VII. CONCLUSION

  • The framework uses Wan 2.2 hypothetical views, ORB-SLAM3 sparse geometry, a measured reference for metric scale, and goal-biased RRT independently of the generated camera trajectory.Generated video is used as virtual sensing for pre-motion geometric planning from one RGB observation.Source: §VII · p. 7
  • Across the 19-scene horizon study, planning-feasible scenes rise from 13/19 to 15/19 and successful RRT trials from 62/90 to 75/90.Five of six physical quadrotor trials also traverse the gate using paths computed before takeoff.Source: §VII · p. 7
  • Across 15 P40 reconstructions, matched structural landmarks show approximately 0.30 m mean discrepancy, while the median additional global scale factor is 0.995×.In S06, corrected dense VDA+SLAM geometry closes an aperture even though sparse SLAM supports 5/5 RRT trials.Source: §VII · p. 7, §VI · p. 7
  • The authors identify broader surface-level evaluation, more environments and robot platforms, and online verification against new physical observations as future work.Online verification could accept, revise, or reject imagined geometry as real observations arrive.Source: §VII · p. 7
Loading 2610.13849v1…