Source-linked AI summary
RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation
Xiaolei Lang, Ze Kang, Zehao Huang, Naiyan Wang
TL;DR
Sparse-view novel view synthesis must combine geometric grounding from observed images with generative priors for unobserved regions, but existing bridges expose generation to lossy scene projections and decouple feedback. RoGe instead queries an implicit reconstruction with target rays to condition video diffusion directly, trains both modules jointly, and outperforms several baseline families on DL3DV while improving temporal consistency.
Problem
Sparse-view novel view synthesis lacks a unified way to preserve observed geometry while generating realistic content in occluded or unobserved regions.
Method
RoGe jointly trains a feed-forward implicit reconstruction model and video diffusion model, using ray-queried per-view geometric features without an explicit 3D intermediate.
Results
RoGe surpasses reconstruction-based, generation-based, and hybrid methods on DL3DV in visual quality, geometric consistency, camera controllability, and temporal consistency.
Takeaways & Limitations
Ray-queried implicit features outperform raw reconstruction tokens and rendered RGB as conditioning, while joint training brings further gains.
Takeaways & Limitations
The method normalizes translations using a scale derived from the maximum translation magnitude in the sparse input views and sets the first view as the world frame.
Abstract
from arXiv · showhide
Novel view synthesis from sparse inputs requires both geometric grounding from the observed views and generative priors of unobserved regions, motivating recent hybrid methods that combine reconstruction and generation. However, existing methods bridge the two with rendered images or explicit 3D representations such as point maps or 3D Gaussians. Generation is thus conditioned on a lossy and imperfect projection of the scene, inheriting its errors, and reconstruction receives no signal from generation to correct them. We present RoGe, an end-to-end unified reconstruction and generation framework that removes this explicit bridge. It targets roaming within a scene anchored by sparse views: given a few posed images and a camera trajectory, it synthesizes a temporally coherent video along that trajectory. From the sparse input views, RoGe builds an implicit scene representation with a feed-forward reconstruction model, and queries it with target camera rays to obtain per-view geometric features. These features are injected into a video diffusion model as conditioning, without any 3D intermediate. Both modules are trained jointly, so the generation objective directly shapes its own geometric conditioning. We conduct experiments on DL3DV, where RoGe outperforms reconstruction-based, generation-based, and hybrid baselines on image-level metrics and video-level temporal consistency. Ablations confirm that ray-queried implicit features outperform both raw reconstruction tokens and rendered RGB as conditioning, and that joint training brings further gains.
1 INTRODUCTION
RoGe addresses sparse-view novel view synthesis by jointly reconstructing implicit geometry and generating unseen views, avoiding an explicit 3D bridge. Its ray-conditioned video diffusion framework targets temporally coherent, controllable roaming and outperforms reconstruction, generation, and hybrid approaches.
- Motivation: Sparse-view roaming requires realistic, temporally coherent novel views that remain consistent with a few posed observations and a specified camera trajectory.Large scene regions may be occluded or entirely unobserved, making the task inherently ill-posed.
- Existing approaches: Reconstruction methods provide camera control and multi-view consistency but struggle with unseen regions, while generation methods handle large viewpoint changes but lag in geometric consistency.The two paradigms therefore offer complementary strengths for sparse-view synthesis.
- Existing approaches: Existing hybrid methods either extend reconstruction with generated views or condition generation on rendered or projected 3D outputs, leaving reconstruction and generation decoupled.Generated views can be inconsistent with inputs, while projected scene representations are lossy and pass their errors to generation.
- RoGe: RoGe connects a feed-forward implicit reconstruction model to video diffusion through per-view geometric features queried along camera rays, without an explicit 3D intermediate.The two modules are trained end-to-end so generation can guide the learned geometric representation.
- Results: On DL3DV, RoGe surpasses reconstruction-based, generation-based, and hybrid methods on visual quality, geometric consistency, camera controllability, and video-level temporal consistency.The reported evaluation covers both image-level and video-level metrics.
2 RELATED WORK
Related work spans scene reconstruction, camera-controllable video generation, and hybrid reconstruction-generation systems. RoGe differs by grounding video generation in geometry recovered from sparse posed views rather than geometry imagined from a single image or text prompt.
- Scene Reconstruction: Classical SfM uses per-scene optimization, whereas feed-forward reconstruction enables efficient single-pass multi-view reconstruction under sparse inputs.Both approaches primarily output point clouds, which are insufficient for high-fidelity novel view synthesis.
- Video Generation: Video generation models support open-domain synthesis and camera-controllable videos from single images or text prompts along user-specified trajectories.This line of work provides generative priors and trajectory control but is distinct from reconstruction grounded in observed sparse views.
- Reconstruction and Generation: Hybrid methods align video diffusion with feed-forward 3D models in feature space, often targeting world modeling from a single image or text prompt with imagined geometry.These systems differ from RoGe's sparse posed-view setting, where geometry can be recovered from observations.
3 METHOD
RoGe combines feed-forward implicit reconstruction with video diffusion for sparse-view novel-view synthesis, using target-ray geometric features as conditioning without an explicit 3D intermediate. Its hybrid representation preserves unordered context views while retaining temporal structure for the generated video.
- Implicit Reconstruction: RoGe reconstructs an implicit scene representation from sparse posed observations and queries it with target camera rays for geometry-aware conditioning.The representation uses intermediate reconstruction tokens rather than depth maps, point clouds, or 3D Gaussians.
- End-to-End Training: RoGe trains reconstruction and generation end-to-end so geometric features are jointly optimized for video synthesis.The reconstruction branch uses rendering supervision, while the generation branch consumes ray-queried features along the target trajectory.
- Implicit Reconstruction: Ray tokens preserve dense correspondence between latent spatial locations and camera rays while querying shared scene tokens through ray–scene cross-attention.The resulting ray tokens are reshaped into spatial feature maps and can also be decoded to RGB for reconstruction supervision.
- Reconstruction-Generation Fusion: A hybrid latent representation independently encodes sparse context images while preserving temporal continuity for the target video.Concatenation and masking preserve permutation invariance for observations while retaining the causal video VAE’s temporal prior for generated views.
- Reconstruction-Generation Fusion: The diffusion transformer receives appearance, explicit camera-ray, and implicit geometric conditions fused with noisy video latents.These conditions are denoted C = {Y, P, G}, with geometry aligned to the latent resolution and temporal organization.
4 EXPERIMENTS
RoGe is evaluated on diverse datasets and against reconstruction, generation, and combined NVS baselines using image- and video-level metrics. Results include DL3DV and ScanNet evaluations, with ablations testing geometric conditioning and joint training.
- Experimental Setup: RoGe is implemented with a camera-conditioned VGGT reconstruction network, ray-map encoder, cross-attention blocks, and Wan2.1-I2V-14B-480P as the video-generation backbone.Training uses six context views and 81 target views, with reconstruction and generation resolutions of 288 × 504 and 192 × 336, respectively.
- Experimental Setup: Training spans ARKitScenes, BlendedMVS, DL3DV, MVS-Synth, and WildRGB-D, while quantitative evaluation uses 10 DL3DV test sequences.The datasets cover indoor and outdoor scenes, object- and scene-level data, street and aerial views, and real and simulated environments.
- Experimental Setup: RoGe is compared with reconstruction-only, generation-only, and reconstruction-generation-combined methods on image and video synthesis.Image metrics include PSNR, SSIM, LPIPS, and DreamSim; video metrics include TSED, MEt3R, FID, FVD, and camera-trajectory APE RMSE.
- Experimental Results and Analysis: On DL3DV, RoGe is evaluated for synthesized images, with comparisons reported in Table 1 and Figure 2.The evaluation contrasts frame-by-frame methods with video methods that generate non-context frames in 81-frame chunks.
- Experimental Results and Analysis: RoGe synthesizes geometrically consistent, visually high-quality videos on DL3DV and ScanNet for both interpolation and extrapolation trajectories.ScanNet extrapolation uses novel trajectories independent of the original ones.
- Ablation Studies: Ablations show that camera-conditioned geometric features resolve frame mismatch, while ray-queried features and joint training improve the results.Raw VGGT patch tokens perform worse than no geometry, camera conditioning provides a modest gain, and joint training further improves all metrics.
5 CONCLUSION
RoGe unifies implicit geometric reconstruction and video generation without an explicit 3D intermediate. Joint training supports temporally coherent novel-view videos with geometric consistency and visual fidelity.
- RoGe injects per-view implicit geometric features from a feed-forward reconstruction model into a video generation model.The framework tightly couples reconstruction and generation without explicit 3D representations.
- End-to-end joint training enables information exchange between reconstruction and generation.The generative module can guide geometric representation learning directly.
- RoGe synthesizes temporally coherent videos along arbitrary camera trajectories with geometric consistency and visual fidelity, including unseen regions.
- Experiments on DL3DV show that RoGe surpasses state-of-the-art methods.