PaperlayerView exact revision on arXiv

Source-linked AI summary

Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective

Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, Chuanxia Zheng, Andreas Geiger, Jianfei Cai, Jia-Wang Bian, Bohan Zhuang

arXiv:2604.14025v1cs.CVcs.AIcs.GR

TL;DR

Classical 3D reconstruction often depends on slow per-scene optimization, limiting scalability and real-time use. This survey organizes feed-forward 3D reconstruction by five model-design problems and finds that benchmark choices reveal a need for standardized scene-complexity quantification and clearer reporting of geometric diversity.

  • Problem

    Classical 3D reconstruction relies on heavy, slow per-scene optimization, creating a need for more efficient and generalizable paradigms.

  • Method

    The survey uses a problem-driven taxonomy spanning five research directions, representations, benchmarks, datasets, applications, and future directions.

  • Results

    The survey identifies benchmark-dependent performance trends and data-driven needs for standardized scene-complexity quantification and clearer geometric-diversity reporting.

  • Takeaways & Limitations

    A problem-driven perspective clarifies shared design concerns and provides a roadmap for understanding progress and future directions in feed-forward 3D reconstruction.

  • Takeaways & Limitations

    Feed-forward reconstruction remains challenged by the trade-off between reconstructing and generating content under occlusion or sparse sampling.

Abstract

from arXiv · show

Reconstructing 3D representations from 2D inputs is a fundamental task in computer vision and graphics, serving as a cornerstone for understanding and interacting with the physical world. While traditional methods achieve high fidelity, they are limited by slow per-scene optimization or category-specific training, which hinders their practical deployment and scalability. Hence, generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years. By learning a model that maps images directly to 3D representations in a single forward pass, these methods enable efficient reconstruction and robust cross-scene generalization. Our survey is motivated by a critical observation: despite the diverse geometric output representations, ranging from implicit fields to explicit primitives, existing feed-forward approaches share similar high-level architectural patterns, such as image feature extraction backbones, multi-view information fusion mechanisms, and geometry-aware design principles. Consequently, we abstract away from these representation differences and instead focus on model design, proposing a novel taxonomy centered on model design strategies that are agnostic to the output format. Our proposed taxonomy organizes the research directions into five key problems that drive recent research development: feature enhancement, geometry awareness, model efficiency, augmentation strategies and temporal-aware models. To support this taxonomy with empirical grounding and standardized evaluation, we further comprehensively review related benchmarks and datasets, and extensively discuss and categorize real-world applications based on feed-forward 3D models. Finally, we outline future directions to address open challenges such as scalability, evaluation standards, and world modeling.

Authors

The paper is authored by researchers from Zhejiang University, Nanyang Technological University, Monash University, ETH Zurich, and the University of Tübingen. Weijie Wang, Qihang Cao, and Sensen Gao share equal contribution, while Jia-Wang Bian and Bohan Zhuang are corresponding authors.

  • The authors are Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, Chuanxia Zheng, Andreas Geiger, Jianfei Cai, Jia-Wang Bian, and Bohan Zhuang.
  • Weijie Wang, Qihang Cao, and Sensen Gao share equal contribution, while Jia-Wang Bian and Bohan Zhuang are corresponding authors.
  • The authors represent Zhejiang University, Nanyang Technological University, Monash University, ETH Zurich, and the University of Tübingen and its Tübingen AI Center.

1 Introduction

The introduction highlights map, pose, constraints, and 3D priors, alongside quality, consistency, and robustness.

  • The introduction identifies map, pose, constraints, and 3D priors as key considerations.
  • It also emphasizes quality, consistency, and robustness.

5 Visual Augmentation

This section concerns visual augmentation, including pre-trained guidance for pose-free reconstruction.

  • 5 Visual Augmentation: Visual augmentation is discussed through pre-trained guidance for pose-free reconstruction.

3 Representation Compaction

The supplied passage situates representation compaction within a survey of core 3D representations and feed-forward networks that generate them. It also places this analysis alongside five challenge-driven research axes, dataset and benchmark reviews, and practical applications.

  • 3 Representation Compaction: The survey introduces core 3D representations, including NeRF, 3D Gaussian Splatting, and Pointmap, alongside other representations.These representations are presented as part of the survey’s overview.
  • 3 Representation Compaction: Feed-forward networks are analyzed in terms of how they generate the surveyed 3D representations.The figure caption explicitly connects the representations with feed-forward network generation.
  • 3 Representation Compaction: The survey organizes subsequent analysis around five axes: feature enhancement, geometry awareness, model efficiency, data or visual augmentation, and temporal modeling.These axes frame the broader methodological analysis beyond the representation overview.
  • 3 Representation Compaction: The survey also reviews datasets and benchmarks and discusses practical applications such as autonomous driving and robotics.The caption identifies these reviews and applications as later components of the survey.

1 Introduction

This survey presents a problem-driven perspective on feed-forward 3D scene modeling, organizing advances around five research directions rather than output representations. It also reviews benchmarks, applications, and future challenges to provide an end-to-end view of the field.

  • Motivation: Feed-forward 3D modeling maps images and optional auxiliary signals to explicit or implicit 3D representations in a single forward pass.This avoids test-time scene optimization and enables faster inference and amortization across scenes.
  • Taxonomy: The survey organizes methods by five problem-driven directions: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware models.These directions emphasize functional roles and design trade-offs rather than categorizing methods by mesh, SDF, NeRF, 3DGS, or Pointmap outputs.
  • Scope: The survey covers representations, datasets and benchmarks, real-world applications, and future directions in a structured, end-to-end panorama.Its application coverage includes autonomous driving, robotics, scene understanding, SfM and SLAM, video generation, and visual localization.
  • Benchmarks: The benchmark review categorizes datasets as geometry-oriented or visual-oriented and highlights the need to quantify scene complexity and report geometric diversity more clearly.Geometry-oriented examples include DTU, ScanNet, and Replica, while visual-oriented examples include NeRF-Synthetic, RealEstate10K, and DL3DV.
  • Future Directions: Future directions span benchmark rigor, model efficiency, scalable scene representations, world models, unified perception and reconstruction, and other open questions.The survey frames feed-forward 3D reconstruction as an active frontier despite its progress from a research concept into a practical technology.

2 Problem Formulation

Generalizable feed-forward 3D models reconstruct scenes from input images in one forward pass without per-scene optimization. Their formulation typically combines image encoding, feature refinement, scene decoding, optional rendering, and joint training on large-scale scene data.

  • 2 Problem Formulation: The model reconstructs a 3D scene from input images in a single forward pass, without optimizing separately for each scene.Existing approaches generally use an encoder, decoder, and optional renderer for novel-view synthesis.
  • 2 Problem Formulation: Input images Ii ∈ R^H×W×3 may be accompanied by corresponding camera poses P∗.
  • 2 Problem Formulation: The encoder Φimage converts the input images into implicit feature maps F = {Fi}K.
  • 2 Problem Formulation: Feature enhancement modules can refine the implicit maps into F′ before the decoder produces the 3D scene representation G.The feature maps use spatial downsampling factor s and feature dimension C.
  • 2 Problem Formulation: The decoder Ψpred maps implicit features to scene representations G, including Gaussian primitives when the representation is 3D Gaussian Splatting.
  • 2 Problem Formulation: In 3D Gaussian Splatting, each primitive is defined by a mean position, covariance matrix, opacity scalar, and color vector, with N total primitives.These quantities are denoted µj ∈ R^3, Σj ∈ R^3×3, αj ∈ R, and cj ∈ R^3, respectively.
  • 2 Problem Formulation: Models are trained once on large collections of scenes and reused at test time, producing representations such as depth maps, voxels, SDFs, meshes, pointmaps, NeRFs, or 3DGS.An optional renderer takes the predicted representation and novel camera poses Pnovel as input.
  • 2 Problem Formulation: The encoder, decoder, and learnable renderer are jointly optimized with a weighted sum of losses, whose active terms depend on the representation and available supervision.Typical geometric losses constrain G against ground-truth annotations G∗ through pointmap regression, depth supervision, and normal consistency.

3 Representations

Feed-forward 3D reconstruction uses diverse representations, including neural fields, explicit Gaussian primitives, pointmaps, and geometry or appearance-aware learned representations. These representations differ in structure and rendering mechanisms, while feed-forward methods aim to predict them directly from images without per-scene optimization.

  • NeRF: NeRF represents scenes as continuous functions mapping 3D position and viewing direction to color and density, rendered through differentiable volume compositing.Its strengths include photorealistic novel-view synthesis, complex-occlusion handling, and view-dependent effects, but original NeRF requires lengthy per-scene optimization and dense MLP queries.
  • NeRF: Feed-forward NeRF methods condition radiance fields on image features to generalize across scenes without test-time retraining.PixelNeRF extracts feature volumes from one or more input views and samples features at projected 3D query locations during ray rendering.
  • 3D Gaussian Splatting: 3DGS models scenes as anisotropic 3D Gaussians with position, covariance, opacity, and view-dependent color, rendered through differentiable visibility-aware splatting.It typically converges within minutes, renders in real time, and supports high-fidelity interactive applications, but depends on precise cameras and SfM-derived initialization.
  • 3D Gaussian Splatting: Feed-forward 3DGS methods predict Gaussian primitives directly from images, moving Gaussian reconstruction beyond per-scene optimization toward faster and more generalizable deployment.pixelSplat uses an encoder–decoder to transform multi-scale image features into pixel-aligned Gaussian parameters in a single forward pass.
  • Pointmap: Pointmaps represent scenes as dense 2D fields of 3D points, establishing one-to-one mappings between image pixels and scene geometry.DUSt3R produces corresponding pointmaps for paired images, with camera-pose transformations relating their coordinates.
  • Alternative representations: Alternative feed-forward representations include geometry-only models such as occupancy, SDFs, and meshes, alongside joint geometry-appearance models such as light fields, texture fields, and triplanes.Occupancy Networks learn continuous binary occupancy functions, while DeepSDF regresses continuous signed distance functions for high-resolution shape reconstruction without discretization artifacts.

4 Research Directions · 4.1 Feature Enhancement

The paper organizes feed-forward 3D reconstruction research around five representation-agnostic problems, with feature enhancement improving implicit features through architectures, cross-view fusion, and visual foundation models. This section emphasizes that feature quality, multi-view coherence, and transferred 2D priors are central to reconstruction accuracy and generalization.

  • 4 Research Directions: The taxonomy organizes methods into five directions: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware models.The taxonomy cuts across NeRF, 3DGS, and Pointmap output representations, categorizing methods by the core challenge they address.
  • 4.1 Feature Enhancement: Feature enhancement targets implicit feature quality because these features directly affect 3D scene decoding, rendering accuracy, and model generalization.The section groups feature enhancement into architectures, cross-view fusion, and visual foundation model integration.
  • 4.1.1 Architectures: Encoder architectures evolved from CNN-based models toward ViTs, Mamba-based networks, and pretrained visual backbones that inject richer visual and geometric knowledge.LRM and Instant3D encode images into feature tokens before transformer decoders unproject them into triplane feature grids, while Gamba models reconstruction as sequential prediction with linear token-length scalability.
  • 4.1.1 Architectures: Architectural designs address cross-view inconsistency and long-sequence complexity through visibility prediction, context-aware feature mixing, state-space models, and persistent scene memory.NeuRay focuses on visible points, C3-GS mixes spatial dimensions and scales, Long-LRM maintains long-range consistency without pure-transformer quadratic complexity, and PreF3R incrementally updates global memory.
  • 4.1.2 Cross-View Fusion: Cross-view fusion establishes feature correspondences and spatial relationships across input images to form coherent, geometrically consistent 3D scenes.AttnRend uses a multi-view ViT with attention across views and camera poses, while geometry-first methods directly regress dense coordinates and camera geometry.
  • 4.1.2 Cross-View Fusion: Geometry-first and memory-based fusion methods improve reconstruction over sparse or long inputs by predicting coordinates, camera geometry, sliding-window consistency, or persistent scene representations.DUSt3R jointly predicts dense pointmaps and relative camera poses; WinT3R trades minimal latency for stronger global consistency; PreF3R progressively refines features with variable-length inputs.
  • 4.1.3 Integration of Visual Foundation Models: Visual foundation models transfer knowledge from large 2D datasets to improve 3D reconstruction generalization and data efficiency.DUSt3R uses CroCo for feature correspondences, Mono3R fuses monocular priors with multi-view stereo features, and Feat2GS maps pretrained 2D features to 3DGS parameters through a lightweight decoder.
  • 4.1.3 Integration of Visual Foundation Models: Foundation-model guidance can recover plausible structure when visual evidence is weak by enriching implicit features or adding language-derived semantics.Mono3R improves regions with weak or unreliable multi-view correspondence, while CATSplat uses vision-language guidance to compensate for missing visual information.

4.2 Geometry Awareness · 4.3 Model Efficiency

The section frames geometry awareness as essential for accurate feed-forward 3D reconstruction and surveys explicit geometric aggregation, surface reasoning, pose-free inference, and pretrained geometric guidance. It then presents model efficiency as addressing computational bottlenecks through feature-efficient processing and compact explicit representations.

  • 4.2.1 Explicit Geometric Aggregation: Geometry-aware pipelines improve reconstruction by incorporating geometric relationships across views rather than relying solely on ambiguous 2D image features.These mechanisms include cost volumes, correspondence constraints, surface-aware modeling, and geometry-guided Gaussian prediction.
  • 4.2.1 Explicit Geometric Aggregation: Explicit aggregation constructs and propagates geometric evidence through cost volumes, feature similarity, surface representations, and hybrid Gaussian architectures.Representative approaches span plane-sweep cost volumes, stereo correspondence, surface-aware SDF modeling, and triplane-Gaussian or MVS-Gaussian representations.
  • 4.2.2 Post Refinement: Post-refinement methods improve complex geometry by hierarchically generating, adaptively distributing, graphically grouping, and locally refining Gaussian primitives.These methods use coarse-to-fine generation, pixel-level alignment, complexity-aware Gaussian counts, Gaussian graphs, and pooling or densification.
  • 4.2.3 Pose-Free Reconstruction: Pose-free reconstruction jointly infers scene geometry and camera parameters from uncalibrated images, using shared volumes, dense pointmaps, canonical coordinates, or reprojection constraints.This direction removes explicit pose requirements while retaining geometric consistency and photometric fidelity.
  • 4.2.4 Pre-trained Geometric Guidance: Pretrained monocular models provide depth, normals, and optical-flow cues that help feed-forward systems bypass the cold-start problem of learning scene geometry.Depth-guided methods connect pretrained geometric priors with multi-view estimation, Gaussian rendering, and single-view reconstruction.
  • 4.3 Model Efficiency: Model-efficiency research targets slow optimization, heavy generalizable models, real-time deployment, and memory-limited settings through feature efficiency and explicit representation compaction.Figure 5 compares memory consumption, Gaussian counts, and inference time across varying input views.
  • 4.3.1 Feature Efficiency: Feature-efficient methods reduce unnecessary multi-view computation through depth-guided and project-aware sampling, informative ray selection, dynamic opacity adjustment, and efficient token processing.VGGT-based approaches further use token merging, quantization, sparse attention, and KV-cache budgeting for constrained deployment.
  • 4.3.2 Representation Compaction: Representation-compaction methods reduce explicit Gaussian complexity by message passing, pooling, pruning, splitting, and pixel-wise triplet fusion.GGN, PixelGaussian, and FreeSplat++ compact or refine Gaussian representations while preserving scene structure.

4.4 Augmentation Strategies · 4.5 Temporal-aware Models

Augmentation strategies expand training distributions or enrich visual representations to address sparse inputs, inaccurate poses, and limited training diversity. Temporal-aware models extend feed-forward reconstruction to dynamic scenes through online, offline, interactive, and task-specific approaches.

  • 4.4.1 Data Augmentation: Data augmentation improves generalization by introducing novel views, simulated camera motions, geometric variations, and large-scale procedurally generated scenes.Puzzles synthesizes posed video-depth data, while MegaSynth generates hundreds of thousands of non-semantic 3D scenes.
  • 4.4.2 Visual Augmentation: Visual augmentation integrates diffusion or generative priors to generate novel views, guide optimization, or enhance pseudo-observations while reducing artifacts and computational cost.MVSplat360 injects Gaussian-rendered features into Stable Video Diffusion to produce photorealistic, 3D-consistent views.
  • 4.4.2 Visual Augmentation: Hybrid geometry-preserving and generative strategies achieve zero-shot adaptation to in-the-wild dynamic scenes and outperform existing methods on dynamic video-based NVS.The passage attributes these gains to combining reconstruction that preserves geometry with strong generative priors.
  • 4.5.1 Online Streaming: Online temporal-aware models update scene representations causally per frame, enabling real-time, open-ended 3D and 4D reconstruction with motion and appearance updates.StreamSplat estimates per-Gaussian velocities and deformations from short temporal cues without external calibration or test-time optimization.
  • 4.5.3 Interactive Modeling: Interactive models reconstruct geometry and visual features offline, then support force injection, content editing, and material adjustment through real-time physical simulation.This paradigm provides immediate feedback on physically plausible results.
  • 4.5.4 Specialized Tasks: Specialized temporal-aware models target application-specific goals including dynamic removal from static mapping and unified reconstruction with tracking.DAS3R suppresses moving objects to reconstruct clean static backgrounds from dynamic videos, while St4RTrack combines reconstruction and tracking.

5 Datasets and Benchmarks

This section organizes representative datasets by purpose, scale, source type, and scene category, and reviews the metrics and benchmark settings used to evaluate feed-forward 3D reconstruction and view synthesis. It covers image synthesis, camera pose, pointmap, and dynamic point tracking evaluation, including a standardized DTU three-view comparison.

  • Evaluation Metrics: Novel view synthesis is commonly evaluated with PSNR, SSIM, and LPIPS, which measure generated-image quality from different perspectives.PSNR denotes Peak Signal-to-Noise Ratio, SSIM denotes Structural Similarity Index, and LPIPS denotes Learned Perceptual Image Patch Similarity.
  • Datasets: Representative datasets are categorized as visual-oriented, geometry-oriented, or mixed, with scale, source type, scene category, and representative training or evaluation methods reported.Scene categories include objects, indoor scenes, outdoor scenes, and mixed scenes; sources are real or synthetic.
  • Evaluation Metrics: Camera pose evaluation uses RTA, RRA, and AUC, while pointmap evaluation uses accuracy, completeness, and Chamfer distance.RTA and RRA measure relative translation and rotation errors, whereas AUC summarizes accuracy across angle-error thresholds.
  • Evaluation Metrics: Dynamic point tracking is evaluated with OA, σvis_avg, and AJ, covering occlusion prediction, tracking within a pixel threshold, and combined occlusion-prediction accuracy.AJ integrates occlusion and prediction accuracy into a comprehensive score.
  • Benchmarks: The DTU benchmark comparison fixes the input to three views for all methods and evaluates early NeRF-based approaches on a relatively small dataset.The section compiles results across multiple benchmarks and datasets, including DTU 3-view and RealEstate10K 2-view novel view synthesis settings.

6 Applications

Feed-forward 3D reconstruction supports scalable, single-pass scene understanding across autonomous driving, robotics, semantic modeling, vision-language systems, SfM/SLAM, and video generation. These applications emphasize real-time efficiency, geometric and semantic consistency, temporal awareness, and downstream physical-world interaction without per-scene optimization.

  • 6.2 Robotics: Robotics applications use feed-forward representations for manipulation and navigation, providing dense geometry, semantics, large-scale maps, and temporally consistent scene memory.Examples include TSDF-based 6-DoF grasp detection, dynamic Gaussian object models, vision-language navigation memory, and GS–mesh hybrids for sim-to-real transfer.
  • 6.1 Autonomous Driving: Autonomous-driving methods target large-scale dynamic reconstruction from sparse views while satisfying low-latency and temporal-consistency requirements.Hierarchical voxel-to-Gaussian pipelines address city-scale static scenes, while 4D Gaussian methods model moving objects; other systems use geometric and depth priors for sparse-camera robustness.
  • 6.3 Semantic 3D Understanding: Feed-forward reconstruction integrates vision-language semantics into 3D representations for open-vocabulary, part-aware, temporally consistent scene understanding from sparse or unposed views.Methods embed 2D semantic features into 3D Gaussians and use segmentation consistency, feature aggregation, Transformer encoding, or semantic-to-geometry regularization.
  • 6.5 Vision-Language Models: Feed-forward 3D pipelines enhance vision-language models by injecting geometric priors or explicit 3D representations for spatial and spatio-temporal reasoning.Approaches use visual–geometry foundation models, spatial branches, structured point clouds, and camera poses to provide spatial cues.
  • 6.4 SfM and SLAM: Differentiable feed-forward architectures unify SfM and SLAM by directly predicting geometry and camera poses, combining global reconstruction accuracy with real-time mapping efficiency and robust trajectory estimation.SfM systems replace incremental optimization with end-to-end global reconstruction, while SLAM systems maintain global map consistency in real time and incorporate sensor fusion or factor-graph optimization.
  • 6.6 Others: Feed-forward 3D reconstruction injects geometric priors into video generation to improve cross-frame consistency, novel-view realism, and physical plausibility.The combination spans reconstruction-enhanced video generation and video generation-based scene reconstruction, producing videos, explicit 3D/4D representations, or geometry-aware world models.

7 Future Directions

Future research should develop scalable representations and efficient deployment systems, strengthen benchmark data and evaluation, and clarify how feed-forward reconstruction should balance generalization, adaptation, generation, and world modeling.

  • Evaluation standards: Standardized benchmarks should vary viewpoint and context difficulty while jointly measuring accuracy, latency, memory, and energy for fairer deployment comparisons.Avoiding selective view sampling is also necessary to prevent unfair evaluation advantages.
  • Scalable representations: Future architectures must support large scenes, long sequences, and high-resolution representations through hierarchical priors, hybrid representations, level-of-detail pipelines, and visibility-aware acceleration.These directions target scalability as computational and memory costs grow super-linearly with global attention and dense 3D processing.
  • Efficient deployment: Deployment requires inference and memory optimization through mixed precision, operator fusion, CUDA graphs, out-of-core scheduling, compression, and hardware-aware execution.Quantization, auto-tuned kernels, and heterogeneous execution across edge devices are proposed to reduce latency and improve utilization.
  • Scalable representations: New representations should be designed specifically for generalizable feed-forward prediction, because per-scene optimized representations may fail to preserve global coherence and fine-grained structure.Directions include explicit-free novel-view generation and strong latent representations that maintain temporal and spatial consistency.
  • World modeling: Future models should clarify reconstruction–generation trade-offs, combine generalization with lightweight scene tuning, and enable action-conditioned incremental updates with persistent long-horizon memory.Video world models inject geometric reasoning through pose conditioning, geometry-grounded latent alignment, and persistent spatial memory.

8 Conclusion

The survey reviews feed-forward 3D reconstruction as a scalable alternative to classical per-scene optimization, emphasizing direct single-pass prediction from input images. It organizes the field around shared challenges in feature enhancement, geometry awareness, and model efficiency.

  • 8 Conclusion: Feed-forward 3D reconstruction predicts 3D representations directly from input images in a single forward pass.This paradigm addresses the scalability and efficiency limitations of classical per-scene optimization.
  • 8 Conclusion: The survey systematically reviews feed-forward 3D reconstruction through a problem-driven taxonomy.The taxonomy highlights shared core challenges across the field.
  • 8 Conclusion: The highlighted challenges include feature enhancement, geometry awareness, and model efficiency.These challenges form part of the survey’s shared problem set for feed-forward 3D reconstruction.
Loading 2604.14025v1…