PaperlayerView exact revision on arXiv

Source-linked AI summary

Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective

Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, Chuanxia Zheng, Andreas Geiger, Jianfei Cai, Jia-Wang Bian, Bohan Zhuang

arXiv:2604.14025v1cs.CVcs.AIcs.GR

TL;DR

Feed-forward 3D reconstruction offers an alternative to slow per-scene optimization, but representation-based organization can obscure methods’ differing design goals. This survey provides a problem-driven synthesis and reports benchmark trends that point to standardized scene-complexity quantification and clearer geometric-diversity reporting [24] [25].

  • Problem

    Existing scene-reconstruction methods often require slow per-scene optimization, while representation-based surveys can obscure distinct design goals, motivating scalable feed-forward methods and a problem-oriented synthesis.

    Source: §1 · pp. 5–6

  • Method

    The survey synthesizes feed-forward 3D work through five problem-driven design directions and reviews representations, datasets, benchmarks, applications, and future directions.

    Source: §1 · p. 6

  • Results

    Its benchmark compilation reveals performance trends across method categories and motivates standardized scene-complexity quantification and clearer reporting of geometric diversity [24] [25].

    Source: §1 · p. 6

  • Takeaways & Limitations

    The problem-driven view exposes shared concerns across methods, including sparse-observation robustness, geometric fidelity, computational efficiency, and temporal coherence.

    Source: §8 · p. 38

    For feed-forward models spanning reconstruction and generation, it remains difficult to determine when to reconstruct versus generate, particularly under occlusion or sparse sampling.

    Source: §7.6 · p. 38

Abstract

from arXiv · show

Reconstructing 3D representations from 2D inputs is a fundamental task in computer vision and graphics, serving as a cornerstone for understanding and interacting with the physical world. While traditional methods achieve high fidelity, they are limited by slow per-scene optimization or category-specific training, which hinders their practical deployment and scalability. Hence, generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years. By learning a model that maps images directly to 3D representations in a single forward pass, these methods enable efficient reconstruction and robust cross-scene generalization. Our survey is motivated by a critical observation: despite the diverse geometric output representations, ranging from implicit fields to explicit primitives, existing feed-forward approaches share similar high-level architectural patterns, such as image feature extraction backbones, multi-view information fusion mechanisms, and geometry-aware design principles. Consequently, we abstract away from these representation differences and instead focus on model design, proposing a novel taxonomy centered on model design strategies that are agnostic to the output format. Our proposed taxonomy organizes the research directions into five key problems that drive recent research development: feature enhancement, geometry awareness, model efficiency, augmentation strategies and temporal-aware models. To support this taxonomy with empirical grounding and standardized evaluation, we further comprehensively review related benchmarks and datasets, and extensively discuss and categorize real-world applications based on feed-forward 3D models. Finally, we outline future directions to address open challenges such as scalability, evaluation standards, and world modeling.

Authors

The paper lists its authors, their institutional affiliations, and contribution and correspondence markers.

Authors

  • The authors are Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, Chuanxia Zheng, Andreas Geiger, Jianfei Cai, Jia-Wang Bian, and Bohan Zhuang.Wang, Cao, and Gao carry the equal-contribution marker; Chen is marked project lead.Source: Authors · p. 2, Affiliations · p. 2

Corresponding Authors

Affiliations

  • The authors are affiliated with Zhejiang University, Nanyang Technological University, Monash University, ETH Zurich, and the University of Tübingen/Tübingen AI Center.The listed countries are China, Singapore, Australia, Switzerland, and Germany, respectively.Source: Affiliations · p. 2

1 Introduction

The survey organizes feed-forward 3D reconstruction around the design problems shaping the field rather than output representation, and reviews its methods, evaluation, applications, and future directions.

1 Introduction

  • Per-scene optimization makes classical methods slow and hard to scale; feed-forward methods map images, optionally with camera or depth cues, to 3D in one pass, while facing fusion, detail, efficiency, and temporal challenges.The survey notes that these challenges drive continued methodological progress.Source: §1 · p. 5
  • Unlike the representation-based organization of the prior survey [20], this survey classifies methods by five problems: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware modeling.It briefly reviews common representations before presenting this problem-driven taxonomy.Source: §1 · pp. 5–6, Figure 1 · p. 5
  • The survey re-evaluates benchmarks and groups datasets by geometry or visual focus; its reported-performance compilation motivates standardized scene-complexity quantification and clearer reporting of geometric diversity [24] [25].Geometry-oriented datasets emphasize point clouds, depth, and pose, whereas visual-oriented datasets prioritize photorealistic view synthesis.Source: §1 · p. 6
  • The survey covers applications in autonomous driving, robotics, scene understanding, SfM and SLAM, video generation, and visual localization, linking feed-forward reconstruction with practical deployment.The passage characterizes these applications as demonstrating increased efficiency and a lower barrier to deployment.Source: §1 · p. 6
  • The survey identifies future directions in benchmark rigor, model efficiency, scalable scene representations, world models, unified perception and reconstruction, and other open questions.It also states that the surveyed references and resources are continuously updated online.Source: §1 · p. 6

2 Problem Formulation

Generalizable feed-forward 3D models reconstruct scenes from input images in a single pass, without per-scene optimization, using an encoder and decoder with an optional renderer. They are trained across collections of scenes and can produce varied 3D representations.

2 Problem Formulation

  • An encoder Φimage maps input images to implicit features, a decoder Ψpred maps those features to a 3D scene representation, and an optional renderer synthesizes novel views.Source: §2 · p. 7
  • Given input images and optionally their camera poses, the encoder extracts a set of implicit feature maps F for downstream scene decoding.Source: §2 · p. 7
  • Feature enhancement modules can refine the implicit feature maps into F′ before the decoder produces the 3D scene representation G.The feature maps are described using a spatial downsampling factor s and feature dimension C.Source: §2 · p. 7
  • With 3D Gaussian Splatting, G comprises Gaussian primitives parameterized by mean position, covariance, opacity, and color.The passage denotes these attributes as μj, Σj, αj, and cj for each of N primitives.Source: §2 · p. 7
  • Training uses a large collection of scenes rather than a separate model for each scene, and methods may output depth maps, voxels, SDFs, meshes, pointmaps, NeRFs, or 3DGS.Source: §2 · p. 7
  • For novel-view synthesis, a renderer takes the predicted scene representation G and novel camera poses Pnovel to synthesize images.Source: §2 · p. 7
  • The encoder, decoder, and any learnable renderer are jointly optimized with weighted loss terms, including geometry supervision and rendered-image photometric or perceptual losses.Geometry losses can constrain G against available annotations, including pointmap regression [3] [26], depth supervision [18], and normal consistency [27] [28]; direct geometry regression without a renderer relies primarily on geometry supervision.Source: §2 · p. 7

3 Representations

The section surveys how feed-forward 3D reconstruction uses several scene representations, from neural fields and Gaussian primitives to pointmaps and other implicit or appearance-aware forms. These methods aim to produce scene reconstructions in a single forward pass rather than through per-scene optimization.

3 Representations

  • Large-scale, multi-scene training amortizes reconstruction so one set of weights can generalize to unseen scenes, which feed-forward systems reconstruct in a single pass without test-time optimization.Confidence-based pruning and merging can further reduce memory use and improve rendering stability.Source: §3 · p. 8

3.1 NeRF

  • NeRF [1] maps 3D position and viewing direction to color and density, then renders pixels by sampling and compositing those values along camera rays.NeRF supports photorealistic novel views, complex occlusions, and view-dependent effects, but the original framework relies on lengthy per-scene optimization, dense MLP queries, and extensive viewpoint calibration; PixelNeRF [13] conditions the field on image features for single-pass reconstruction.Source: §3.1 · pp. 8–9

3.2 3D Gaussian Splatting

  • 3DGS [2] represents scenes as anisotropic Gaussians rendered by visibility-aware splatting, enabling real-time rendering but often requiring precise camera poses, SfM initialization, and substantial memory.Feed-forward methods such as pixelSplat [15] predict pixel-aligned Gaussian parameters directly from image features in one pass, bypassing per-scene optimization and supporting more generalizable reconstruction.Source: §3.2 · pp. 9–10

3.3 Pointmap

  • Pointmaps encode a 3D point for each image pixel; DUSt3R [3] predicts pointmaps and relative camera poses from multiple images in one pass, unifying depth estimation with pose inference.The paired pointmaps express points from one view in the coordinate frame of the other.Source: §3.3 · p. 10

3.4 Others

  • Other feed-forward representations include geometry-only occupancy, SDF, and mesh models, alongside joint geometry-and-appearance forms such as light fields, texture fields, and triplane-based radiance.Occupancy Networks [5] learn continuous occupancy probabilities, while DeepSDF [4] regresses continuous signed distance functions and IM-NET [63] decodes implicit fields conditioned on shape embeddings.Source: §3.4 · p. 10

4 Research Directions

To push feed-forward models toward greater robustness, accuracy, and efficiency, the section organizes research into five problem-driven directions that cut across output representations. It includes representative methods addressing feature enhancement, geometry awareness, model efficiency, augmentation, and temporal modeling.

4 Research Directions

  • The taxonomy groups research into five directions: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware models.These directions target better representations and fusion, geometric priors, lower computation and memory overhead, improved generalization, and dynamic or streaming scenes.Source: §4 · pp. 11–12
  • Figure 2 presents representative NeRF, 3DGS, and Pointmap methods, while traditional feed-forward work also includes voxel, mesh, occupancy, SDF, and other representations.The survey notes that only representative methods are included.Source: Figure 2 · p. 12
  • Methods are categorized by the core challenge they address, so NeRF, 3DGS, and Pointmap approaches can appear in any of the five directions.The taxonomy cuts across output representations rather than grouping methods by their 3D output format.Source: §4 · p. 12

4.1 Feature Enhancement

Feature enhancement improves the implicit features that condition 3D scene decoding, through evolving encoder architectures, geometrically consistent cross-view fusion, and priors transferred from visual foundation models. These approaches include both feature-level refinement and direct geometric prediction or scene-state accumulation.

4.1.1 Architectures

  • Transformer and state-space architectures extend feature extraction from CNN backbones toward global context, single-image 3D decoding, and efficient long-sequence reconstruction.LRM [75] and Instant3D [76] decode ViT tokens into triplanes, while GRM [92] and GS-LRM directly predict Gaussian parameters; GNT uses view and ray transformers, and Long-LRM [33] uses a state-space model to avoid quadratic complexity on long sequences.Source: §4.1.1 · pp. 13–15

4.1.2 Cross-View Fusion

  • Cross-view fusion seeks geometrically consistent features through attention, iterative refinement, and direct prediction of 3D geometry.AttnRend uses attention across views and camera poses; eFreeSplat combines cross-completion pre-training with Gaussian alignment, while iLRM [106] iteratively refines features. Geometry-first methods include DUSt3R [3], which regresses pointmaps and relative poses, and MV-DUSt3R+ [100], which adds reference views and a Gaussian prediction head.Source: §4.1.2 · p. 15
  • Long-sequence methods maintain global consistency by accumulating scene information in memory or processing frames with bounded windows and scalable state representations.PreF3R [102] incrementally updates spatial memory, Spann3R [101] stores point-wise features, and CUT3R maintains a compressed global state that can infer unobserved structures and handle dynamic scenes. WinT3R [104] uses a sliding window; IncVGGT, ZipMap [120], LoGeR [121], tttLRM [122], and VGG-T3 [123] offer further memory-, chunk-, or test-time-training-based approaches to long-context reconstruction.Source: §4.1.2 · pp. 15–16

4.1.3 Integration of Visual Foundation Models

  • Visual foundation approaches enrich reconstruction with monocular priors and pre-trained 2D geometric features, improving details where multi-view correspondence is weak and structural plausibility when visual cues are limited.Mono3R [124] fuses multi-view stereo features with monocular representations; Feat2GS uses a lightweight decoder to map pre-trained 2D features into 3D Gaussian Splatting parameters; CATSplat [94] uses language-derived semantics to compensate for missing visual information.Source: §4.1.3 · pp. 16–17

4.2 Geometry Awareness

Geometry-aware feed-forward reconstruction strengthens geometric reasoning because shape fidelity governs photorealism and multi-view consistency. The section organizes methods around explicit geometric aggregation, post-refinement, pose-free reconstruction, and pre-trained geometric guidance.

4.2.1 Explicit Geometric Aggregation

  • Cost volumes and multi-view cues make geometry explicit, while correspondence-based methods infer structure without requiring dense volume construction.MVSNeRF uses plane-sweep cost volumes, GeoNeRF adds cascaded volumes to guide rendering, and MVSplat [17] localizes Gaussian centers with plane-sweep cues; SRF and GPNR [79] instead use stereo or epipolar correspondence.Source: §4.2.1 · pp. 18–19
  • Surface-aware and feature-similarity methods encode geometry through SDFs, depth distributions, feature matching, or explicit mesh structure.SparseNeuS [65] and VolRecon [66] use surface representations, while ReTR [67] models depth distributions; MatchNeRF and GTA use feature similarity or geometric attention, and RenderFormer [73] reasons over triangle meshes.Source: §4.2.1 · pp. 18–19
  • Gaussian and hybrid aggregation combines multi-view geometric evidence with Gaussian representations for efficient, detailed reconstruction.AGG [77], TGS [78], and LaRa [74] use triplane, Gaussian, or attention-based hybrids; MVSGaussian initializes Gaussian optimization from MVS point clouds, while TranSplat uses depth confidence and monocular depth priors to address limited overlap and unreliable matching.Source: §4.2.1 · p. 19

4.2.2 Post Refinement

  • Post-refinement methods improve Gaussian geometry through hierarchical or adaptive densification, aggregation, pooling, or iterative gradient-based updates.HiSplat [137] generates coarse then fine Gaussians with error-aware refinement; PixelGaussian [139] adapts Gaussian count to local complexity, GGN pools Gaussian graphs, GD densifies in one pass, and G3R adds renderer-based iterative refinement.Source: §4.2.2 · pp. 19–20

4.2.3 Pose-Free Reconstruction

  • Pose-free reconstruction jointly infers geometry and camera parameters from uncalibrated images, avoiding dependence on known poses.LEAP aggregates image features into a shared neural volume, while DUSt3R [3] regresses dense pointmaps; Pow3R can condition on auxiliary camera information, and π3 [153] predicts permutation-equivariant poses and pointmaps.Source: §4.2.3 · p. 20
  • Pose-free Gaussian methods use canonical frames, learned alignment, or joint pose-and-primitive prediction to reconstruct unposed views.NoPoSplat [145] predicts Gaussians in a canonical input-view frame, PF3plat [146] uses pretrained depth and correspondence for coarse alignment, and SPFSplat jointly predicts camera extrinsics and Gaussians with reprojection and rendering losses; YoNoSplat [155] handles posed or unposed image sets.Source: §4.2.3 · pp. 20–21

4.2.4 Pre-trained Geometric Guidance

  • Pre-trained geometric guidance transfers depth, normals, pointmaps, or optical-flow priors into reconstruction to improve geometry and reduce cold-start difficulty.DepthSplat [18] links multi-view depth estimation with 3DGS and uses differentiable rendering as an unsupervised depth objective; PM-Loss [28] regularizes pointmap-derived geometry, while Flash3D [156], Niagara [157], and JointSplat [160] use monocular depth, normals, or optical flow for single-view or feed-forward reconstruction.Source: §4.2.4 · p. 21

4.3 Model Efficiency

Model efficiency addresses the limits of slow per-scene optimization and heavy generalizable models through two directions: improving feature efficiency and compacting explicit 3D representations.

4.3.1 Feature Efficiency

  • Feature-efficient approaches use project-aware sampling to predict a small number of informative ray samples, compress inputs, fuse dense view features into compact latents, or update compact viewpoint tokens.Long-LRM [33] combines token merging with a hybrid Mamba2-Transformer backbone to handle approximately 250k tokens from 32 views before decoding per-pixel Gaussians.Source: §4.3.1 · p. 22
  • VGGT-related work improves deployment efficiency with token merging, quantization, sparse attention, KV-cache budgeting, and geometry-aware cached-token merging.FastVGGT [166], QuantVGGT [167], Sparse VGGT [168], Evict3R [169], LiteVGGT [171], and Speed3R [170] target runtime, memory, accuracy, or scalability; SR3R [165] frames 3D super-resolution as feed-forward mapping from sparse low-resolution views to high-resolution 3D.Source: §4.3.1 · p. 22

4.3.2 Representation Compaction

  • Explicit Gaussian compaction merges, prunes, deduplicates, or bounds splat counts using learned structure and multi-view evidence.GGN pools Gaussian-graph groups; PixelGaussian [139] uses keypoint scoring and deformable attention to guide pruning and splitting; FreeSplat++ [172] deduplicates overlaps and removes floaters, while LongSplat [173] compresses redundancy online. Feature-efficient encoders also use opacity regularization, confidence masks, or post-hoc pruning to bound Gaussian counts.Source: §4.3.2 · p. 23

4.4 Augmentation Strategies

Section 4.4 frames augmentation as two complementary strategies: expanding training data and using generative priors to improve visual outputs. Both aim to address limitations such as sparse inputs, inaccurate poses, and insufficient training diversity.

4.4.1 Data Augmentation

  • Because feed-forward reconstruction is limited by training-data scale and diversity, augmentation expands training with novel views, structures, and synthetic scenes to improve generalization.Puzzles [174] synthesizes posed video-depth data; MegaSynth procedurally generates non-semantic scenes; Aug3D [176] adds structure-from-motion novel views; and MVBoost creates pseudo-ground-truth through generative refinement and consistency checks.Source: §4.4.1 · pp. 23–24

4.4.2 Visual Augmentation

  • MVSplat360, LatentSplat, ProSplat [179], DIFIX3D+ [181], and CogNVS [264] integrate generative models into Gaussian-based or video reconstruction pipelines.Their approaches include diffusion-guided view synthesis, uncertainty-aware latent Gaussians, one-step refinement, artifact suppression, and inpainting hidden pixels; generative priors address artifacts and missing areas, while feed-forward NVS can blur poorly observed regions.Source: §4.4.2 · p. 24
  • The hybrid strategy enables zero-shot adaptation to in-the-wild dynamic scenes and significantly surpasses existing methods on dynamic video-based novel-view synthesis.It combines geometry-preserving reconstruction with strong generative priors.Source: §4.4.2 · p. 25

4.5 Temporal-aware Models

Temporal-aware feed-forward models reconstruct 4D scenes by maintaining geometry and motion consistency across frames. They process time through online streaming, offline sequence or window processing, or interactive user-controlled models, alongside task-specific approaches.

4.5.1 Online Streaming

  • Online models update scene representations frame by frame, supporting real-time, open-ended reconstruction through motion-aware, recurrent-state, causal-Transformer, and long-stream approaches.StreamSplat [182] estimates Gaussian velocities and deformations without external calibration or test-time optimization; DGS-LRM [183] predicts deformable Gaussians and scene flow from monocular video. Cut3R carries geometry across frames to reduce drift and flicker, Stream3R [184] scales causal pointmap prediction to long streams, and LongStream [185] targets stable metric-scale reconstruction over very long sequences.Source: §4.5.1 · p. 25

4.5.2 Offline Processing

  • Offline models process full clips or long windows non-causally, trading latency and memory for globally consistent 4D reconstructions.Methods include per-frame Gaussian generation with temporal attention in L4GM, queryable space-time fields in 4D-LRM [190], and dynamic-video pointmap aggregation in MonST3R [187] and Easi3R [191]. 4DGT [192] uses rolling windows and density control; 4Real-Video-V2 [193] integrates multi-view and temporal cues, while MoVieS [194] shares appearance, depth, and motion encoding. Task-driven variants address sparse-view fusion with MonoFusion [195], egocentric sequence estimation with EgoMono4D [188], and timestamped “bullet time” reconstruction with BTimer [189].Source: §4.5.2 · p. 26

4.5.3 Interactive Modeling

  • Interactive models let users edit content, apply forces, or adjust materials and receive rapid physics-based feedback.PIXIE [197] predicts a material field in one feed-forward pass after offline geometry and feature reconstruction, enabling on-demand simulation on fixed geometry. PhysGM [198] predicts a Gaussian scene and physical attributes from an image or sparse views in one inference, avoiding per-scene optimization and supporting low-latency editing and animation.Source: §4.5.3 · p. 26

4.5.4 Specialized Tasks

  • Specialized feed-forward models target dynamic-object removal or unified reconstruction and tracking rather than universal 4D reconstruction.DAS3R [199] learns a static attribute for each Gaussian to suppress moving objects and reconstruct a clean background. St4RTrack predicts world-space pointmaps and persistent temporal correspondences in one pass, keeping point identities consistent without a separate tracking module.Source: §4.5.4 · p. 27

5 Datasets and Benchmarks

The section categorizes datasets by scene, source, and geometry-versus-visual orientation, then summarizes evaluation metrics and benchmark comparisons for feed-forward 3D reconstruction and view synthesis.

5 Datasets and Benchmarks

  • Datasets are categorized by scene type and real-world or synthetic source, and as visual-oriented, geometry-oriented, or mixed.Geometry-oriented datasets provide ground-truth 3D representations such as point clouds, depth, and camera poses; visual-oriented datasets suit novel-view synthesis and photorealistic rendering.Source: §5 · p. 27, Table 1 · p. 28
  • Evaluation covers image quality, camera pose, point maps, and dynamic point tracking using task-specific metric families.Novel-view synthesis uses PSNR, SSIM, and LPIPS; pose uses RTA, RRA, and AUC; point maps use accuracy, completeness, and Chamfer distance; tracking uses OA, σvis_avg, and AJ.Source: §5 · pp. 27, 29, 30
  • MuRF leads DTU 3-view NVS, while iLRM [106] leads RealEstate10K 2-view NVS among 34 compiled methods.All DTU methods use three input views; RealEstate10K is substantially larger, typically uses two views, and includes early NeRF-based and recent 3DGS approaches.Source: Table 2 · p. 29, Table 3 · p. 29, §5 · p. 30
  • Depth-Anything-3, π3, and Map-Anything lead point-map estimation on 7-Scenes, NRGBD, and ETH3D, respectively, in both sparse- and dense-view settings.The evaluation uses three input views for sparse-view and ten for dense-view settings, reporting Acc., Comp., NC, and Overall.Source: §5 · p. 30

6 Applications

Feed-forward 3D reconstruction extends single-pass scene modeling across driving, robotics, semantic understanding, SfM/SLAM, video generation, and other applications. Across these settings, methods use geometric and semantic representations or priors to support efficient reconstruction, reasoning, mapping, and synthesis.

6.1 Autonomous Driving

  • Autonomous-driving methods address dynamic city-scale reconstruction, sparse camera coverage, and temporal consistency using hierarchical voxel-to-Gaussian or direct 4D Gaussian pipelines, with task-specific systems for downstream driving needs.WorldSplat also integrates diffusion for novel-view video synthesis; other methods use geometric priors and depth features to stabilize inference with sparse or limited-overlap cameras.Source: §6.1 · p. 31

6.2.1 Manipulation.

  • For manipulation, GraspNeRF predicts TSDFs for 6-DoF grasp detection, while Gaussian-based methods represent object geometry and dynamics for multibody, bimanual, and action-oriented tasks.Other applications include reinforcement learning, open-vocabulary instruction following, and personalized real-to-sim-to-real navigation.Source: §6.2.1 · p. 32

6.2.2 Navigation.

  • For navigation, feed-forward 3D maps support planning and localization through semantic panoramic memory, GS–mesh simulation memory, change-aware map refinement, and real-time image-goal localization.These systems target vision-and-language queries, sim-to-real transfer, object-centric navigation, and adaptive planning.Source: §6.2.2 · p. 32

6.3.1 Semantic Feature Fields

  • Semantic-feature methods embed 2D semantic features into 3D Gaussians while jointly reconstructing geometry and appearance, and use consistency, feature aggregation, or semantic-to-geometry regularization to improve sparse-view generalization.Related work also supports open-vocabulary, part-aware, and temporally consistent scene understanding without per-scene optimization.Source: §6.3.1 · pp. 32–33

6.3.2 Spatial Reasoning

  • For spatial reasoning, methods inject geometric foundation-model priors into vision-language models or retain explicit 3D cues, supporting structural reasoning, multi-view consistency, and egocentric temporal reasoning.Examples include VGGT- or CUT3R-based visual encoders and video aligned with point clouds and camera poses.Source: §6.3.2 · p. 33

6.4 SfM and SLAM

  • Feed-forward SfM pipelines jointly infer geometry and camera parameters, while SLAM methods reconstruct local geometry and maintain global map consistency in real time; newer systems add sensor fusion, Gaussian decoders, or factor graphs.Together, these approaches point toward unified differentiable systems combining global reconstruction accuracy with real-time mapping and robust trajectory estimation.Source: §6.4 · p. 33

6.5 Video Generation

  • Video-generation methods either use reconstruction as a structural prior to improve video consistency or distill video-model priors into explicit 3D/4D scenes and geometry-aware world models.The two directions include cyclic generation–reconstruction refinement and geometry-grounded controls, spatial memory, or geometric rewards for long-horizon scene evolution.Source: §6.5 · pp. 33–34

6.6 Others

  • Other applications include panoramic reconstruction using spherical geometry and Gaussian representations, and feed-forward localization through direct pose prediction or learned scene correspondences.Table 4 describes pointmap estimation on 7-Scenes, NRGBD, and ETH3D using accuracy, completeness, normal consistency, and their mean as overall error, under 3- and 10-view settings.Source: Table 4 · p. 31, §6.6 · p. 34

7 Future Directions

Future work must improve evaluation, efficiency, and scalable representations while integrating feed-forward reconstruction into richer, interactive world models and perception systems. Key open questions concern how to balance reconstruction with generation and generalization with scene-specific adaptation.

7.1 Rigorous Benchmarks

  • Future benchmarks should pair high-fidelity 3D ground truth with video, stratify viewpoint and context difficulty, and standardize protocols to prevent selective view sampling from distorting comparisons.Current benchmarks often lack ground-truth point clouds and vary in difficulty or use fixed input views, limiting reliable and fair evaluation.Source: §7.1 · p. 35

7.2 System Efficiency

  • Future systems must balance accuracy, throughput, latency, and hardware constraints while scaling to large scenes, long sequences, and high-resolution representations.Promising directions include hierarchical priors, hybrid explicit–implicit representations, level-of-detail pipelines, and occupancy- or visibility-aware acceleration. Mixed precision, operator fusion, out-of-core scheduling, compression, hardware-aware optimization, and benchmarks measuring accuracy, latency, memory, and energy can support deployment.Source: §7.2 · pp. 35–36

7.3 Scalable Representations

  • New representations designed for feed-forward reconstruction should scale across environments while preserving local detail and global coherence.Representations adapted from per-scene optimization can struggle to maintain coherence and fine detail at scale; hierarchical or compositional structures are proposed to address this trade-off.Source: §7.3 · p. 36

7.4 World Models

  • Video and 3D world models offer complementary strengths, but unifying their representations, enabling action-conditioned updates, and maintaining persistent memory remain open challenges.Video models scale and capture rich appearance and motion, but latent world states are difficult to query or constrain; explicit 3D states support editing and simulation but lag in visual richness and filling unseen regions.Source: §7.4 · p. 37

7.5 Unified Perception and Reconstruction

  • Unified perception could combine visual, text, audio, and sensory inputs with language-guided scene editing to create richer, semantically aware, interactive 3D scenes.Natural-language commands could let users edit reconstructed scenes, while interactive world models are proposed as components for embodied intelligence, robotics, and simulation.Source: §7.5 · pp. 37–38

7.6 Open Questions

  • Open questions include how to guide generative priors without sacrificing reconstruction fidelity and whether practical deployment should permit lightweight per-scene refinement.Research should clarify the reconstruction–generation trade-off under occlusion or sparse sampling and balance generalization with scene-specific tuning.Source: §7.6 · p. 38

8 Conclusion

The survey organizes feed-forward 3D reconstruction around shared design challenges and reviews its benchmarks and applications. It identifies evaluation, scalability, geometric consistency, and integration with generative and semantic modeling as open directions.

8 Conclusion

  • Feed-forward reconstruction predicts 3D representations from images in one pass, addressing the scalability and efficiency limits of classical per-scene optimization.The taxonomy covers feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware modeling, alongside concerns such as sparse-observation robustness, geometric fidelity, and temporal coherence.Source: §8 · p. 38
  • The survey separates benchmarks into geometry-oriented and visual-oriented categories, calling for rigorous protocols that distinguish geometric accuracy from perceptual fidelity.This distinction is intended to avoid overfitting to view-synthesis metrics.Source: §8 · p. 38
  • The reviewed applications span autonomous driving, robotics, scene understanding, and video generation, positioning feed-forward reconstruction as a broader 3D perception backbone for spatial intelligence.Source: §8 · p. 38
  • Future research directions include standardized benchmarks, scalable scene representations, improved geometric consistency, and deeper integration with generative modeling and semantic understanding.Source: §8 · p. 38

Related papers

Loading 2604.14025v1…