PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction

Ziren Gong, Guo Chen, Yongjia Li, Yihua Shao, Fabio Tosi, Stefano Mattoccia, Matteo Poggi, Hao Tang, Fei Ma, Shuyan Li, Ziyang Yan, Nicu Sebe, Ling Shao, Jianfei Cai, Qi Tian, Ming-Hsuan Yang

arXiv:2609.39960v1cs.CV

TL;DR

Dynamic reconstruction must address non-rigid motion and temporal coherence, while evaluation protocols for temporal consistency and topology changes remain unstandardized. This survey organizes NeRF and 3DGS methods and examines their benchmark performance; results vary across tasks and methods. It provides a structured reference and roadmap for future work.

  • Problem

    High-fidelity 4D reconstruction must handle non-rigid-motion ambiguities and preserve long-range spatio-temporal coherence, while standardized evaluation of temporal coherence and topology changes remains lacking.

    Source: §1 · p. 2, §5.2.3 · p. 11

  • Method

    The survey organizes NeRF and 3DGS approaches by their dynamic-scene modeling paradigms, reviews datasets and metrics, and analyzes representative methods and open challenges.

    Source: §1 · p. 2, §4 · p. 7, §7 · p. 14

  • Results

    Benchmark findings vary by task: external priors improve 3DGS results, while efficient and motion-aware methods perform strongly on different NeRF-DS measures.

    Source: §5.3 · p. 12

  • Takeaways & Limitations

    The survey serves as a structured reference and roadmap, highlighting feed-forward modeling, hybrid representations, generative priors, and applications in embodied AI and dynamic scene understanding.

    Source: §7 · p. 14

    Standardized protocols remain lacking for evaluating temporal coherence, topology changes, and long-term consistency.

    Source: §5.2.3 · p. 11

Abstract

from arXiv · show

4D scene reconstruction aims to recover the evolving geometry, appearance, and motion of dynamic environments from visual observations. Despite substantial progress in neural scene representations, reconstructing dynamic scenes remains challenging due to non-rigid motion, occlusions, temporal inconsistencies, and the trade-offs between reconstruction fidelity and computational efficiency. Recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have introduced diverse approaches to representing and reconstructing dynamic scenes, yet their relationships, underlying design choices, and evaluation protocols remain fragmented. In this paper, we present a unified perspective on 4D scene reconstruction, organizing existing methods around their scene representations, temporal modeling strategies, reconstruction pipelines, and optimization objectives. Through this framework, we examine how different design choices affect geometric fidelity, appearance consistency, motion representation, and computational efficiency. We further consolidate commonly used datasets and evaluation metrics, identify limitations in current experimental practices, and discuss open challenges in reconstructing complex, dynamic real-world environments. By connecting methodological developments with their underlying assumptions and evaluation evidence, this work provides a structured foundation for understanding existing approaches and identifying future research directions. An evolving collection of relevant papers and resources is available at https://github.com/ZiyangYan/Awesome-4D-Scene-Reconstruction.

1 INTRODUCTION

Dynamic reconstruction must recover coherent geometry, appearance, and motion despite non-rigid change and temporal ambiguity. The survey organizes a fragmented NeRF- and 3DGS-based field around scene-specific optimization and reviews representative methods, benchmark performance, and open challenges.

1 INTRODUCTION

  • Classical SfM and MVS established static reconstruction pipelines, while NeRF and 3DGS introduced learned continuous representations for modeling geometry and appearance.Classical methods rely on hand-crafted features and geometric assumptions; 3DGS uses anisotropic Gaussians and tile-based rasterization for real-time rendering.Source: §1 · pp. 1–2
  • High-fidelity 4D reconstruction must handle non-rigid-motion ambiguities, long-range spatio-temporal coherence, and the separation of changing radiance from geometric deformation.These challenges make dynamic environments substantially more complex than static scene modeling.Source: §1 · p. 2
  • The survey addresses a fragmented field by focusing on scene-specific 4D NeRF and 4DGS, reviewing representative benchmark performance and open challenges.Existing surveys primarily emphasize static neural rendering, leaving systematic review of scene-specific dynamic reconstruction lacking.Source: §1 · p. 2
  • Per-scene optimization anchors reconstruction to scene observations, mitigating the generalization gap of feed-forward models that can hit a fidelity ceiling on out-of-distribution scenes.Feed-forward approaches offer rapid offline-style inference but rely on pretrained dataset biases.Source: §1 · p. 2

2 PRELIMINARIES

Dynamic reconstruction has progressed from geometry-based pipelines to learning-based methods, NeRF, and 3D Gaussian Splatting, each with distinct efficiency and representation trade-offs. This survey compares implicit and explicit radiance-field approaches and broadens prior survey coverage with datasets, protocols, and quantitative performance analysis.

2.1 History of Dynamic Scene Reconstruction

  • Early geometric methods track per-frame geometry or fuse depth into canonical volumes, while dynamic NeRFs model non-rigid motion but remain costly to train and render.Dynamic Gaussian Splatting enables real-time rasterization and compact geometry-appearance representation, but may require many primitives and high memory.Source: §2.1 · pp. 2–3

2.2 Representing Radiance Fields

  • NeRF implicitly maps 3D coordinates and viewing direction to color and density for volume rendering, whereas 3DGS explicitly represents scenes with anisotropic Gaussians projected and alpha-blended.NeRF applies positional encoding to capture high-frequency details; 3DGS Gaussians are initialized from a sparse SfM point cloud.Source: §2.2 · p. 3

2.3 Comparison with Existing Surveys

  • The survey covers both NeRF- and GS-based methods, datasets, evaluation protocols, and quantitative comparisons; Zhu et al. [49] covers 52 methods and 10 datasets versus 102 and 22 here.The comparison also identifies scene types omitted by Zhu et al. [49], including autonomous driving, and notes the absence of a unified taxonomy and comprehensive evaluation in that survey.Source: §2.3 · p. 3

3 DYNAMIC NEURAL RADIANCE FIELDS

NeRF-based 4D reconstruction methods encode dynamics through deformation, unified space-time fields, structured feature volumes, or external temporal cues. These choices trade modeling flexibility and coherence against computation, memory, reliance on priors, and robustness to complex motion.

3.1 Preliminaries of 4D NeRF

  • NeRF-based 4D methods are grouped into deformation fields, implicit 4D primitives, feature volumes, and temporal priors, which guide temporal modeling in different ways.Deformation methods map observations to a canonical representation; implicit primitives add time directly, feature volumes factorize space-time, and temporal-prior methods add auxiliary supervision.Source: §3.1 · p. 4

3.4 4D Feature-Volume-Based Methods

  • Feature-volume methods factorize space-time into lower-dimensional structures, including planes, tensor components, and grids, to reduce computation and memory while retaining expressiveness.HexPlane and K-Planes use orthogonal planes; these factorizations can trade memory usage against spatial resolution.Source: §3.1 · p. 4, §3.4 · p. 5

3.2 Deformation-Field-Based Methods

  • Deformation-field methods map time-varying coordinates into a canonical space, but a single continuous field can struggle with large displacements and topology changes.Localized canonical subspaces address articulated or interacting-object motion, although optimization can be expensive and sensitive to initial trajectories or poses.Source: §3.2 · p. 5

3.3 Implicit 4D Primitive-Based Methods

  • Implicit 4D primitives directly map space, time, and viewing direction to density and color, avoiding canonical decomposition while requiring careful temporal sampling and greater computation.Motion fields, depth priors, and object-centric scene graphs address flow, monocular ambiguity, and multiple moving agents, but remain sensitive to errors, occlusions, or unpredictable motion.Source: §3.1 · p. 4, §3.3 · p. 5

3.4 4D Feature-Volume-Based Methods

  • Temporal interpolation and frequency-aware models support smooth transitions, while hash grids and multimodal features scale to large or sparse-observation scenes with task-specific signals.High-frequency motion and abrupt topology changes may be blurred; hash grids can collide, and specialized priors can limit generalization across scenarios.Source: §3.4 · pp. 5–6

3.5 Temporal-Prior-Based Methods

  • Temporal-prior methods use external cues such as flow, physical constraints, or sequential updates to regularize dynamic reconstruction without explicitly parameterizing time in the radiance field.Incremental streaming updates can accumulate misalignments over long sequences; statistical regularization may suppress geometric detail, and LiDAR constraints depend on motion-estimation quality.Source: §3.5 · pp. 6–7

4 DYNAMIC 3D GAUSSIAN SPLATTING

The section organizes 3DGS-based dynamic reconstruction around three ways of parameterizing time: explicit 4D primitives, deformation fields, and frame-wise training. Each offers different trade-offs in flexibility, efficiency, and temporal coherence.

4 DYNAMIC 3D GAUSSIAN SPLATTING

  • The three paradigms—explicit 4D primitives, deformation fields, and frame-wise training—differ in how they parameterize scene evolution and trade flexibility, efficiency, and temporal coherence.Explicit methods treat time as a geometric dimension; deformation methods learn transformations; frame-wise methods optimize discrete states.Source: §4 · p. 7

4.1 Fundamentals of 4DGS

  • A 4D Gaussian represents space and time in its mean and covariance, then is sliced at a queried time into a 3D Gaussian for rendering.The rotation couples spatial and temporal dimensions; the resulting formulation integrates with standard splatting while maintaining temporal consistency.Source: §4.1 · p. 7

4.2 Explicit 4D Primitive-Based Methods

  • Explicit 4D primitive methods embed time directly in Gaussian parameters and use conditional slicing for temporal consistency and efficient real-time rendering, at the cost of greater per-primitive memory.Each Gaussian requires a 4D mean and a 4 × 4 covariance matrix; the approach is suited to indoor dynamic scenes with dense multi-view capture and real-time rendering requirements.Source: §4.2 · p. 7

4.3 Deformation-Field-Based Methods

  • Deformation-field methods predict attribute offsets from a canonical 3D Gaussian set, improving parameter efficiency and supporting non-rigid motion and complex appearance changes.MLP-based prediction may introduce rendering jitter, and its additional inference can limit rendering speed.Source: §4.3 · p. 8

4.4 Frame-Wise Training Methods

  • Frame-wise training handles topology changes through dynamic Gaussian-set updates, but storage grows linearly with sequence length and long-range coherence is limited without inter-frame coupling.Keyframe interpolation and flow-supervised losses can partially address these issues, with approximation errors or reliance on external priors.Source: §4.4 · pp. 9–10

5 PERFORMANCE EVALUATION

The evaluation section consolidates representative datasets and benchmark results, prioritizing cross-verified papers that use consistent evaluation protocols.

5 PERFORMANCE EVALUATION

  • The comparison prioritizes papers with consistent benchmarks and a shared evaluation framework across sources.This priority addresses cases where papers do not release code or specific configurations.Source: §5 · p. 10
  • The section organizes quantitative data from papers using a common evaluation protocol and cross-verifies the results.Source: §5 · p. 10
  • It summarizes representative datasets and compares novel-view-synthesis and geometric-reconstruction results, highlighting first-, second-, and third-best results.Source: §5 · p. 10

5.1 Benchmark Datasets

The benchmark overview distinguishes datasets by capture conditions, scene content, and temporal scope, noting what each supports and where evaluation remains constrained.

5.1.1 Synthetic vs. Real-world Data

  • Synthetic datasets offer clean articulated or deformable scenes, while real-world datasets capture complex lighting, calibration errors, and dynamic backgrounds but often lack accurate ground truth.Source: §5.1.1 · p. 10

5.1.2 Monocular vs. Multi-view Capture

  • Monocular benchmarks suit methods relying on strong priors but can have scale ambiguity and limited spatial coverage; multi-view capture supports fidelity and view-consistency evaluation.Sparse multi-view datasets reflect real-world constraints and support evaluation of view-sparse reconstruction.Source: §5.1.2 · p. 10

5.1.3 Short-term vs. Long-term Horizons

  • Short-horizon datasets support deformation and short-term consistency evaluation but may miss long-range dynamics; long-horizon driving datasets support temporal-consistency and prediction evaluation.Long-horizon benchmarks also have sparse viewpoints and noisy annotations.Source: §5.1.3 · p. 10

5.1.4 Human-Centric vs. General Dynamic Scenes

  • Human-centric datasets support articulated-motion and fine-grained geometry evaluation, while general-dynamics datasets cover diverse elements but challenge motion decomposition and scene understanding.Source: §5.1.4 · p. 10

5.1.5 RGB-only vs. Multimodal Datasets

  • RGB-only datasets support appearance evaluation but face depth ambiguity and lighting sensitivity; multimodal datasets support geometric-accuracy evaluation under challenging conditions.DyCheck's smartphone LiDAR also supports evaluation in lightweight capture settings.Source: §5.1.5 · p. 10

5.2 Evaluation Metrics

The evaluation framework covers rendering quality, geometric accuracy, and computational efficiency, while noting that commonly used metrics leave important aspects of dynamic scenes unmeasured.

5.2 Evaluation Metrics

  • The evaluation framework assesses rendering quality, geometric accuracy, and computational efficiency using widely adopted metrics.It also organizes representative datasets by benchmark properties and research focus.Source: §5.2 · p. 10, Table 4 · p. 11

5.2.1 View Synthesis Metrics

  • PSNR, SSIM, and LPIPS measure frame-wise appearance, but none explicitly models temporal consistency or dynamic geometric accuracy.PSNR measures pixel-wise error; SSIM evaluates structure and contrast; LPIPS uses deep feature representations.Source: §5.2.1 · p. 10

5.2.2 Geometric and Spatiotemporal Metrics

  • Chamfer Distance, F-Score, and RMSE assess geometry, but are sensitive to point density, threshold selection, or outliers and may miss topology changes or perceptual quality.Figure 5 shows qualitative reconstruction point maps and depth for NeRF-style methods on NuScenes.Source: §5.2.2 · pp. 10–11, Figure 5 · p. 11

5.2.3 Efficiency and Practicality

  • FPS measures inference speed, but alone omits memory consumption and scalability; standardized protocols for temporal coherence, topology changes, and long-term consistency remain an open challenge.Despite these limitations, the metrics remain widely adopted and provide a common comparison basis.Source: §5.2.3 · p. 11

5.3 Novel View Synthesis

Across the reported novel-view benchmarks, feature-volume approaches often lead, while results also highlight the value of structured priors and motion-aware representations. Performance patterns vary by dataset and method family.

Neu3D and D-NeRF

  • Under protocols and, Neu3D recent hybrid approaches improve over earlier implicit models, with 4D feature-volume architectures performing strongly; on D-NeRF, these methods consistently achieve state-of-the-art across diverse dynamic benchmarks.Source: §5.3 · pp. 11–12

Neu3D

  • On Neu3D, recent 3DGS methods improve PSNR; ADC-GS achieves the best SSIM, while ED-3DGS yields strong LPIPS scores.Table 6 reports Neu3D 3DGS-style results.Source: §5.3 · p. 12, Table 6 · p. 12

NeRF-DS

  • On NeRF-DS, EfficientGS achieves the best results, while motion-aware methods perform strongly on dynamic regions.The passage attributes this pattern to deformation-field representations that decouple temporal motion from canonical geometry.Source: §5.3 · p. 12, Table 8 · p. 12

Waymo

  • On Waymo, general-purpose methods often ghost or collapse on distant or fast-moving objects; EmerNeRF and OmniRE use scene-flow, feature, or semantic priors for more robust reconstruction.The cited discussion associates EmerNeRF's scene flow and DINOv2 features with lighting robustness, and OmniRE's category-specific priors with physically plausible human structure.Source: §5.3 · p. 12

5.4 Geometric Reconstruction

Geometric evaluation combines surface extraction and point-based metrics, with NuScenes results showing advantages for LiDAR-supervised methods and distinct strengths in global error and local depth accuracy.

NuScenes

  • OmniRE yields superior local depth accuracy and robustness to outliers, suggesting deformation-field representations capture fine geometric details and preserve structural integrity.The evaluation uses surface extraction and point-based metrics, including Chamfer Distance, RMSE, and F1-score at a 5 cm threshold.Source: §5.4 · pp. 12–13

5.5 Model Efficiency

Efficiency comparisons reveal a speed–memory trade-off between 3DGS-style and NeRF-style methods.

5.5 Model Efficiency

  • 3DGS-style methods render faster but require larger model capacity, whereas NeRF-style methods use memory more efficiently at the cost of slower rendering.DevRF and StreamRF are the most efficient NeRF-style methods; 4DRotorGS attains the highest FPS among 3DGS methods.Source: §5.5 · p. 13

6 FUTURE PROSPECTS

Future work emphasizes sparse-input reconstruction, hybrid and generative representations, and controllable scenes; applications span embodied AI, dynamic-scene understanding, and human-centered settings.

6.1 Technical Prospects

  • Feed-forward models use large reconstruction models and transformer architectures to infer 4D representations, enabling near-real-time reconstruction from sparse inputs and linking reconstruction with spatio-temporal understanding.The field is shifting from per-scene optimization toward direct inference.Source: §6.1 · p. 13
  • Hybrid explicit–implicit designs combine efficient rasterization with latent fields for non-Lambertian effects and temporal topology changes, addressing scaling challenges for open-world dynamic scenes.The passage contrasts explicit methods’ memory limitations with implicit methods’ difficulty supporting real-time interaction.Source: §6.1 · p. 13
  • Generative priors may support plausible scene completion, motion prediction, and view synthesis when inputs are sparse or degraded.This direction could shift reconstruction toward predictive and generative frameworks.Source: §6.1 · p. 13
  • Future 4D systems are expected to let users manipulate scenes, edit object behavior, and simulate alternatives, supporting applications such as simulation, digital twins, and content creation.The passage describes a shift from passive reconstruction toward interaction and controllability.Source: §6.1 · pp. 13–14

6.2 Application Prospects

  • Application prospects include richer embodied-AI training environments, dynamic-scene navigation and editing, and human-motion modeling for AR/VR, telepresence, healthcare, and digital twins.Dynamic 4D scenes provide interaction signals for perception, planning, and interaction in complex settings.Source: §6.2 · p. 14

7 CONCLUSION

The survey synthesizes NeRF- and 3DGS-based 4D reconstruction methods, datasets, and evaluation metrics. It discusses open challenges and future directions as a roadmap for the field.

7 CONCLUSION

  • The survey analyzes NeRF- and 3DGS-based approaches in terms of efficiency, scalability, and temporal coherence, and summarizes representative methods, datasets, and evaluation metrics.It also discusses open challenges, future directions, and applications in embodied AI and dynamic-scene understanding.Source: §7 · p. 14

Related papers

Loading 2609.39960v1…