EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

Jiawei Yang; Boris Ivanovic; Or Litany; Xinshuo Weng; Seung Wook Kim; Boyi Li; Tong Che; Danfei Xu; Sanja Fidler; Marco Pavone; Yue Wang

Source-linked AI summary

EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, Yue Wang

arXiv:2311.02077v1Nov 3, 2023cs.CV

TL;DR

Dynamic driving-scene reconstruction must recover expansive 4D structure from sparse observations without costly annotations or pretrained dynamic-object models. EmerNeRF uses self-supervised static-dynamic fields, an emergent flow field, and lifted foundation-model features, achieving state-of-the-art results across reconstruction, synthesis, and flow tasks while introducing a 120-sequence benchmark.

Problem
Learning 4D representations of dynamic scenes from sparse observations is challenging, while semantic features and Transformer positional embeddings complicate consistent 3D scene comprehension.
Method
EmerNeRF learns static and dynamic neural fields with self-supervision, uses induced scene flow for temporal feature aggregation, and lifts foundation-model features into 4D.
Results
EmerNeRF achieves state-of-the-art performance in sensor simulation, dynamic-scene reconstruction, novel view synthesis, and scene flow estimation.
Takeaways & Limitations
The method provides a self-supervised representation for highly dynamic driving scenes and supports benchmarking with 120 challenging Waymo-derived sequences.
Takeaways & Limitations
The work focuses on autonomous-driving data from public spaces, leaving neighborhood and population diversity difficult to manage comprehensively.

Abstract

from arXiv · show

We present EmerNeRF, a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes. Grounded in neural fields, EmerNeRF simultaneously captures scene geometry, appearance, motion, and semantics via self-bootstrapping. EmerNeRF hinges upon two core components: First, it stratifies scenes into static and dynamic fields. This decomposition emerges purely from self-supervision, enabling our model to learn from general, in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field from the dynamic field and uses this flow field to further aggregate multi-frame features, amplifying the rendering precision of dynamic objects. Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to represent highly-dynamic scenes self-sufficiently, without relying on ground truth object annotations or pre-trained models for dynamic object segmentation or optical flow estimation. Our method achieves state-of-the-art performance in sensor simulation, significantly outperforming previous methods when reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual foundation model features into 4D space-time and address a general positional bias in modern Transformers, significantly boosting 3D perception performance (e.g., 37.50% relative improvement in occupancy prediction accuracy on average). Finally, we construct a diverse and challenging 120-sequence dataset to benchmark neural fields under extreme and highly-dynamic settings.

1 INTRODUCTION

EmerNeRF addresses sparse-observation dynamic-scene reconstruction with a self-supervised 4D neural representation that separates static and dynamic content and estimates scene flow. It is evaluated on a challenging 120-sequence driving benchmark designed to emphasize diverse, highly dynamic conditions.

EmerNeRF lifts 2D visual foundation-model features into 4D space-time and addresses detrimental Transformer positional-embedding patterns.
Dynamic-scene 4D reconstruction is difficult because mobile robots obtain sparse, transient observations, often seeing each space-time point only once.
EmerNeRF decouples static and dynamic scene components and predicts 3D scene flows entirely through self-supervision.Its density-regularized hybrid representation uses multi-timestep observations for static elements and flow-based feature aggregation for dynamic elements.
The approach constructs a hybrid static-dynamic world representation while generating dynamic-object density only where points intersect dynamic objects.
The NOTR benchmark contains 120 diverse driving sequences selected to support focused evaluation under challenging dynamic conditions.The sequences come from Waymo Open Dataset sensor data and include varied lighting, weather, and exposure conditions.

2 RELATED WORK

Prior dynamic-scene methods often depend on synchronized multi-view data, multiple traversals, ground-truth annotations, or pretrained optical-flow models. EmerNeRF instead targets accurate 4D reconstruction from a single traversal using self-supervision and intrinsic flow predictions.

Earlier dynamic-view-synthesis methods rely on multiple synchronized videos from different viewpoints, limiting their use in autonomous driving and robotics.
D2NeRF, NeuralGroundplans, and SUDS each retain important constraints involving outdoor-scene capacity, synchronized views, pretrained flows, or expensive flow-based losses.
EmerNeRF reconstructs an accurate 4D scene representation from a single-traversal self-driving-vehicle log.
EmerNeRF uses and refines intrinsic flow predictions, forming a self-improving loop without pretrained flow models.
Unlike approaches requiring ground-truth annotations for dynamic objects, EmerNeRF achieves high-fidelity simulation results through self-supervision.

3 SELF-SUPERVISED SPATIAL-TEMPORAL NEURAL FIELDS

EmerNeRF learns a self-supervised 4D representation by separating static and dynamic fields, using an emergent flow field for temporal feature aggregation, and lifting semantic features while removing positional-embedding artifacts.

3 SELF-SUPERVISED SPATIAL-TEMPORAL NEURAL FIELDS: EmerNeRF uses static, dynamic, and flow fields to represent scene geometry, appearance, motion, and semantics through self-supervision.Its design avoids ground-truth annotations and pre-trained models for dynamic segmentation or optical-flow estimation.
3.1 SCENE REPRESENTATIONS: The static field models time-independent features from x, while the dynamic field models time-varying features from (x, t) and predicts separate densities.Both fields use learnable hash grids followed by lightweight MLPs.
3.1 SCENE REPRESENTATIONS: A density-based renderer combines static and dynamic field outputs from discrete samples along each ray to produce pixel estimates.The rendering weights use per-point density and transmittance.
3.1 SCENE REPRESENTATIONS: Dynamic density regularization encourages the dynamic field to produce density only when necessary, supporting static-dynamic decomposition.The regularizer exploits the predominance of static content in the world.
3.2 SCENE FLOW ESTIMATION: The flow field predicts forward and backward 3D motion, and weighted features from adjacent timesteps are integrated with the current dynamic feature.The aggregation uses weights 0.25, 0.5, and 0.25 for previous, current, and next timesteps, respectively.
3.3 VISION TRANSFORMER FEATURE LIFTING: EmerNeRF lifts 2D foundation-model features into 4D, reconstructs features with shared heads, and separates positional-embedding patterns from PE-free features.The method addresses fixed image-space patterns that break 3D multi-view consistency and impair perception.

4 EXPERIMENTS

The experiments evaluate EmerNeRF on reconstruction, novel view synthesis, scene flow, and few-shot semantic occupancy using the NOTR benchmark and related comparisons. Results show strong reconstruction and flow performance, while positional-embedding decomposition improves foundation-model feature utility.

Benchmark: NOTR contains 120 hand-picked sequences spanning static, dynamic, and diverse driving conditions for evaluating neural fields on challenging autonomous-vehicle data.The benchmark includes Static32, Dynamic-32, and Diverse-56 splits across seven conditions and provides annotations for several driving-perception tasks.
Evaluation protocol: Novel view synthesis evaluation omits every 10th timestep, producing 10% novel temporal views measured with PSNR and SSIM.Scene reconstruction uses all samples in a log for training and probes each method’s upper bound.
Rendering: EmerNeRF consistently outperforms competing methods on dynamic and static scene reconstruction and novel view synthesis.Compared methods include HyperNeRF, D2NeRF, StreetSurf, and iNGP; qualitative comparisons also show stronger preservation of high-frequency details.
Flow estimation: EmerNeRF outperforms NSFP across EPE3D, Acc5, and Acc10 without explicit flow supervision.The reported flow behavior emerges from temporal aggregation during reconstruction optimization.
Foundation-model features: 37.50% average relative improvement in micro-accuracy is obtained by using PE-free DINOv2 features instead of PE-containing counterparts for few-shot occupancy prediction.PE-free DINOv1 features also receive a 24.95% average relative boost in micro-accuracy.
Foundation-model features: Positional-embedding decomposition markedly improves feature synthesis quality, especially for DINOv2.Table 4 compares feature-PSNR for PE-containing and PE-free models.

5 CONCLUSION

The conclusion presents EmerNeRF as a self-supervised 4D representation that jointly models static and dynamic content, flow, and optional semantic features. On challenging driving scenes, it achieves state-of-the-art sensor-simulation performance, while foundation-model features support future few-shot and open-vocabulary capabilities.

Conclusion: EmerNeRF represents geometry, appearance, motion, and semantic features through static and dynamic fields, an emergent flow field, and optional foundation-model feature lifting.Except for foundation-model feature lifting, these tasks are learned self-supervised without object annotations or pre-trained dynamic-segmentation and optical-flow models.
Conclusion: EmerNeRF achieves state-of-the-art sensor-simulation performance on the 120-scene NOTR benchmark, outperforming prior methods on static and dynamic reconstruction, novel view synthesis, and scene flow estimation.The benchmark is a challenging subset of the Waymo Open Dataset.
Future work: Foundation-model features open future directions including few-shot, zero-shot, and auto-labeling through open-vocabulary detection.These are identified as future capabilities enabled or significantly improved by foundation-model features.

ETHICS STATEMENT

The ethics statement focuses on privacy and dataset diversity in autonomous-driving data. Although NOTR diversifies neighborhoods and scenarios within its source data, it remains geographically limited to primarily urban areas in a handful of U.S. cities.

Privacy and diversity: Waymo and NOTR blur faces and license plates, but instance-level privacy protection does not address broader demographic, architectural, geographic, and temporal diversity.The statement emphasizes repeated traversals and coverage across neighborhoods and times of day as additional concerns.
Scope boundary: NOTR primarily represents urban geographies collected from only a handful of cities in the USA.This scope boundary is inherited from the parent Waymo Open Dataset.

A IMPLEMENTATION DETAILS

EmerNeRF is trained end-to-end with sensor-specific supervision, regularization, sampling, and jointly optimized static, dynamic, and flow components.

Sampling: Training uses two proposal models for multi-level sampling and mixes 25% hard-example rays with 75% uniformly sampled rays.Hard examples are selected using periodically recomputed low-resolution color discrepancies.
EmerNeRF trains all components jointly in an end-to-end manner.
Loss functions: The total pixel-ray objective combines RGB, sky, shadow, dynamic-density, cycle-consistency, and semantic-feature losses.The loss is L = Lrgb + Lsky + Lshadow + Lσd(pixel) + Lcycle + Lfeat.
Loss functions: Dynamic regularization penalizes mean dynamic density, encouraging the dynamic branch to generate density only when necessary.
Loss functions: LiDAR supervision combines expected-depth and line-of-sight losses, with free-space and near-surface terms supporting clear static-dynamic decomposition.The line-of-sight loss promotes a unimodal density-weight distribution along each ray.
Training: Dynamic training takes 2 hours without semantic features and 2.25 hours with feature embedding on one A100 GPU.Static training takes 33 minutes without features and 40 minutes with them.

A.2 BASLINE IMPLEMENTATIONS

Baseline implementations adapt HyperNeRF and D2NeRF to the NOTR dataset and standardize parts of their supervision for comparison.

HyperNeRF and D2NeRF use modified official JAX implementations, trained for 100k iterations with batch size 4096.Training and evaluation take approximately 4 hours on 4 A100 GPUs per scene for each model.
Both baselines receive a sky head and the same depth and sky supervision as EmerNeRF to improve comparability.
Figure B.1 samples the NOTR benchmark across static, dynamic, and seven challenging scene categories, with LiDAR visualizations and sky masks.

B NERF ON-THE-ROAD (NOTR) DATASET

NOTR is a 120-sequence Waymo-derived benchmark designed to evaluate neural fields across static, dynamic, and diverse challenging driving conditions.

NOTR contains 120 unique driving sequences: 32 static scenes, 32 dynamic scenes, and 56 scenes across seven challenging conditions.The conditions are ego-static, high-speed, exposure mismatch, dusk/dawn, gloomy, rainy, and nighttime.
The dataset adds bounding boxes for dynamic objects, ground-truth 3D scene flow, and 3D semantic occupancy.These resources support motion understanding and scene comprehension beyond view synthesis.
Static scenes follow the StreetSuRF split, while dynamic and Diverse-56 scenes are selected and organized primarily by lighting or ego-vehicle conditions.
Figure C.1 presents qualitative scene reconstruction comparisons.

C.1 QUALITATIVE RESULTS

Qualitative comparisons report that EmerNeRF produces detailed reconstructions while separating static and dynamic content and generating dynamic scene flows.

EmerNeRF consistently delivers more realistic and detailed renders than previous methods in qualitative reconstruction comparisons.
Unlike HyperNeRF’s composite-only renders, EmerNeRF separates static and dynamic elements while rendering high-fidelity temporal views.
EmerNeRF also introduces dynamic scene-flow generation alongside its reconstruction capabilities.

C.2 ABLATION STUDIES

The ablation study evaluates how EmerNeRF components affect scene reconstruction, novel view synthesis, and scene flow estimation. Combining the tested settings yields the best results, while removing temporal aggregation or freezing temporal feature gradients prevents flow estimation from emerging.

The ablations assess component impacts on scene reconstruction, novel view synthesis, and scene flow estimation.All ablation models are trained for 8k iterations, compared with 25k iterations in the primary experiments.
Removing temporal aggregation or freezing temporal feature gradients negates the emergence of flow estimation ability.
Combining all tested settings yields the best results across the ablation study.

C.3 LIMITATIONS

EmerNeRF has limitations related to sensor effects, camera-pose optimization, geometry–rendering trade-offs, and motion estimation for slow objects. Limited observations and fast ego-motion can exacerbate its difficulty with slow-moving objects.

EmerNeRF does not optimize camera poses and is prone to rolling shutter effects in camera and LiDAR sensors.The authors suggest joint pixel-wise camera-pose optimization and LiDAR rolling-shutter compensation as future directions.
The balance between geometry and rendering quality remains a trade-off requiring further study.
EmerNeRF occasionally struggles to estimate slow-moving-object motion when the ego-vehicle moves fast.The challenge is exacerbated by limited observations.