PaperlayerView exact revision on arXiv

Source-linked AI summary

Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction

Jiaxin Huang, Yuanbo Yang, Bangbang Yang, Lin Ma, Yuewen Ma, Yiyi Liao

arXiv:2601.04090v2cs.CV

TL;DR

Existing 3D scene-generation methods face geometric-quality, optimization-cost, and limited-3D-supervision challenges. Gen3R combines VGGT-derived geometry latents with video-diffusion appearance latents in a jointly generated, aligned space, producing controllable RGB videos and 3D outputs. Experiments report stronger appearance, geometry, camera controllability, and reconstruction robustness than prior approaches and a two-stage alternative.

  • Problem

    3D scene generation is limited by poor geometric structure, high optimization cost, scarce 3D ground truth, and difficult geometry learning from 2D supervision.

  • Method

    Gen3R recasts VGGT as a geometric-latent provider, aligns its latents with video-diffusion appearance latents, and jointly generates both under varied conditioning.

  • Results

    Gen3R outperforms existing 2D- and 3D-based generative methods in appearance, geometry, and camera controllability while improving feed-forward reconstruction robustness.

  • Takeaways & Limitations

    The unified model supports single- or multi-view generation, camera-conditioned or unconstrained operation, RGB videos, point clouds, depth maps, camera parameters, and reconstruction.

Abstract

from arXiv · show

We present Gen3R, a method that bridges the strong priors of foundational reconstruction models and video diffusion models for scene-level 3D generation. We repurpose the VGGT reconstruction model to produce geometric latents by training an adapter on its tokens, which are regularized to align with the appearance latents of pre-trained video diffusion models. By jointly generating these disentangled yet aligned latents, Gen3R produces both RGB videos and corresponding 3D geometry, including camera poses, depth maps, and global point clouds. Experiments demonstrate that our approach achieves state-of-the-art results in single- and multi-image conditioned 3D scene generation. Additionally, our method can enhance the robustness of reconstruction by leveraging generative priors, demonstrating the mutual benefit of tightly coupling reconstruction and generative models.

1. Introduction

Gen3R addresses the difficulty of generating photorealistic, geometrically consistent 3D scenes by combining reconstruction-model geometry priors with video-diffusion appearance priors. Its aligned latent design supports controllable, multi-view-consistent synthesis across flexible input settings.

  • 3D scene generation matters for simulation, gaming, robotics, virtual reality, immersive environments, training data, and creative content design.
  • Existing approaches often suffer from poor geometric structure or high optimization cost despite promising results.
  • Geometry-centric VAEs are difficult to learn because large-scale 3D ground truth is scarce and 2D-only supervision often yields suboptimal geometry and constrained generation quality.
  • VGGT provides a compact token space encoding multi-view geometry, including depth, camera pose, and global structure, motivating its use as a geometric latent provider.
  • Gen3R aligns VGGT-derived geometry latents with video-diffusion appearance latents while keeping them disentangled, then jointly generates controllable video and globally consistent 3D point clouds.
  • The unified framework supports single or multiple views, optional camera cues, feed-forward reconstruction, temporally coherent RGB videos, and globally aligned point clouds.
  • The method combines strong RGB priors with rich geometric priors in a disentangled yet aligned latent space for controllable, multi-view-consistent scene synthesis.

2. Related Work

Related work spans 2D-prior-based generation, feed-forward scene generation, and learned reconstruction. Gen3R differs by integrating feed-forward geometric priors with diffusion generation for coherent scene synthesis and reconstruction error mitigation.

  • 3D Scene Generation from 2D Priors: 2D-prior methods use score distillation, multi-view synthesis, or incremental outpainting to generate 3D content.
  • Feed-Forward 3D Scene Generation: Feed-forward object generation benefits from large-scale 3D datasets, but full-scene generation remains difficult because high-quality scene-level data is scarce.
  • Feed-Forward 3D Scene Reconstruction: Learned reconstruction methods infer geometrically consistent point clouds and increasingly estimate camera parameters, dense geometry, and point tracks jointly.
  • Gen3R: Gen3R combines a feed-forward reconstruction model's geometric prior with a generative diffusion model, enabling coherent 3D scene synthesis from one or two views.
  • Gen3R: The method also performs reconstruction and mitigates errors from the original reconstruction model.

3. Method

Gen3R bridges VGGT reconstruction and video diffusion by jointly generating aligned appearance and geometry latents. These latents decode into controllable RGB videos and globally consistent 3D scene attributes under varied conditioning settings.

  • Unified geometry and appearance latents: Gen3R recasts VGGT as a geometry-latent provider and combines its geometric priors with appearance latents from a pretrained video diffusion model.The method targets high-fidelity scenes with consistent geometry and controllable cameras from one or more images.
  • Geometry representation: VGGT infers geometry tokens from input images and decodes them into point clouds, depth maps, and camera parameters.The reconstruction model uses intermediate transformer tokens and DPT heads for these multimodal geometric predictions.
  • Geometry adapter: An adapter maps VGGT geometry tokens into the video diffusion model’s lower-dimensional latent space and maps generated geometry latents back for decoding.The resulting geometry latents share the diffusion model’s spatial-temporal resolution and feature dimension, enabling joint generation.
  • Latent alignment: The adapter combines reconstruction and distribution-alignment losses, with KL regularization aligning geometry latents to pretrained RGB latents.This alignment addresses unconstrained mapped latents that can hinder diffusion convergence and degrade generation quality.
  • Joint latent generation: A fine-tuned video diffusion model jointly generates concatenated appearance and geometry latents using text, image, mask, and optional camera conditions.Training samples one-view, two-view, or all-frame image conditions, while camera conditions may be dropped for inference flexibility.
  • Decoding and inference: Separate decoders transform the generated latents into photorealistic RGB frames, globally consistent point clouds, depth maps, and camera parameters.Inference supports one-view generation, two-view generation, and feed-forward reconstruction, with or without camera conditions depending on the setting.

4. Experiments

Gen3R is evaluated across 3D generation, reconstruction, appearance, geometry, and camera controllability settings using diverse multi-view datasets and ablations. Results show stronger generation and reconstruction quality, while joint latent generation and distribution alignment improve appearance, geometry, and camera control.

  • Training Details: Experiments use over 300k multi-view consistent scenes spanning indoor, outdoor, driving, object-centric, and synthetic domains.Training combines RealEstate10K, DL3DV-10K, ACID, TartanAir, KITTI-360, Waymo, Co3Dv2, MVImgNet, Virtual KITTI 2, and WildRGB-D.
  • Appearance Generation: Gen3R outperforms or matches most appearance-generation baselines and plausibly completes occluded regions through diffusion-based generation.The comparison includes reconstruction-based, 2D generation, and explicit 3D generation methods under 1-view and 2-view conditions.
  • Geometry Generation: Gen3R clearly outperforms Aether and WVD in Chamfer Distance for both 1-view and 2-view geometry generation settings.It produces more complete globally consistent geometry, while VGGT has higher accuracy but lower completeness and worse Chamfer Distance than Gen3R.
  • Feed-forward 3D Reconstruction: The geometric VAE retains VGGT’s competitive reconstruction performance, and the generative model further improves reconstruction by correcting artifacts such as floaters.The method outperforms Aether and WVD on reconstruction despite a reimplemented WVD using VGGT priors.
  • Ablation Study: Joint RGB-geometry generation outperforms a two-stage RGB-then-VGGT pipeline in appearance, geometry, and camera-control accuracy.Removing the distribution-alignment loss makes geometry latents diverge from appearance latents and significantly degrades convergence, controllability, and generation quality.

5. Conclusion

Gen3R couples feed-forward reconstruction with video diffusion to synthesize high-fidelity scenes, jointly producing RGB videos and globally consistent geometry. Experiments report gains in appearance, geometry, camera controllability, and reconstruction robustness.

  • Gen3R reformulates VGGT as an asymmetric geometry VAE and aligns its latents with a video diffusion model.The framework jointly generates RGB videos and globally consistent 3D geometry.
  • Gen3R outperforms existing 2D- and 3D-based generative methods in appearance and geometry while providing superior camera controllability.
  • Gen3R improves feed-forward reconstruction robustness, illustrating mutual benefits from combining generative priors with geometric foundations.

6. Implementation Details

Gen3R encodes image and geometry conditions into a unified latent representation for joint diffusion. Its geometry adapter converts VGGT tokens into diffusion-compatible geometry latents, while the modified transformer processes appearance and geometry together.

  • Processing Input Conditions: Condition images become appearance latents, while masks and optional camera conditions encode which input views and poses are available.Missing condition images are represented by zeros.
  • Processing Input Conditions: Appearance and geometry latents are fused with masks and concatenated across width to form the unified condition latent.The resulting condition latent is concatenated channel-wise with the noised diffusion latents.
  • Geometry Adapter: The geometry adapter projects VGGT geometry tokens into the video diffusion latent space and decodes generated geometry latents back into geometry tokens.The adapter broadcasts camera tokens spatially and combines them with four other token types.
  • Diffusion Transformer: The diffusion transformer uses 36 input channels and shared RoPE embeddings for corresponding appearance and geometry positions.This supports width-wise concatenation of the two modalities.

7. Additional Comparison Results

Additional evaluations show strong generation across camera-conditioned, camera-free, and out-of-distribution settings, alongside competitive pose estimation and improved reconstruction geometry. Gen3R also trades slight zero-shot accuracy for better completeness and Chamfer distance than VGGT.

  • 3D Generation: Gen3R surpasses existing methods across all appearance metrics and datasets in the 1-view camera-conditioned setting and leads in the 2-view setting.Qualitative results also show closer adherence to camera conditions and better 3D structure.
  • 3D Generation Without Camera Conditions: Gen3R outperforms Aether and WVD in camera-free 3D scene generation on the reported VBench dimensions.The evaluation covers subject, background, aesthetic, imaging-quality, and motion-smoothness scores.
  • Out-of-Distribution Datasets: Gen3R maintains consistent generation performance on unseen LLFF, Mip-NeRF 360, and ScanNet++ scenes excluded from training.
  • Camera Pose Estimation: Gen3R achieves competitive camera pose estimation against VGGT and surpasses Aether on RealEstate10K and WildRGB-D.
  • Geometry Reconstruction: Gen3R produces cleaner reconstructed point clouds than VGGT, while VGGT slightly exceeds it in zero-shot accuracy on ScanNet++.Gen3R achieves better completeness and Chamfer distance, indicating reasonable generalization to unseen scenes.
  • RGB Reconstruction: Wan’s RGB VAE significantly outperforms an RGB head trained on VGGT geometry tokens for RGB reconstruction.The comparison attributes this gap to VGGT’s focus on geometry rather than high-fidelity appearance reconstruction.

8. More Results of Gen3R

Additional visualizations cover camera-conditioned and camera-free generation plus feed-forward reconstruction. Gen3R produces globally consistent, photorealistic scenes across diverse environments and supports controllable or free navigation depending on camera inputs.

  • The additional results visualize generated frames, sequence depth maps, and global point clouds for generation and reconstruction tasks.They include camera-conditioned, camera-free, and feed-forward reconstruction settings.
  • Gen3R handles indoor, outdoor, and object-centric scenes while synthesizing globally consistent and photorealistic 3D content.
  • Camera inputs provide strong controllability, whereas their absence enables free scene navigation.The model supports multiple coherent output modalities across these settings.
Loading 2601.04090v2…