Source-linked AI summary
D$^2$NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, Cengiz Oztireli
TL;DR
The paper addresses monocular-video disentanglement of camera and object motion while recovering the static 3D environment. It introduces a self-supervised radiance-field decomposition with separate dynamic, static, and shadow modeling, and reports superior results across several decoupling, removal, synthesis, and segmentation tasks.
Problem
Monocular dynamic-scene reconstruction is ill-posed, while image-domain solutions lack 3D understanding and existing methods often require additional supervision or restricted scene settings.
Method
D2NeRF uses separate neural radiance fields for dynamic and static content, a skewed-entropy loss for separation, and a shadow field for time-varying shadows.
Results
The method achieves superior results across multiple tasks and datasets, including 3D scene decoupling, clean-background novel-view synthesis, occlusion and shadow removal, and moving-object segmentation.
Takeaways & Limitations
Self-supervised monocular-video reconstruction can decouple dynamic objects and correlated shadows while recovering clean static-background representations.
Takeaways & Limitations
The method relies on accurate camera registration and can incorrectly decouple reflective-surface effects or ambiguous small-range motion as dynamic effects.
Abstract
from arXiv · showhide
Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence. Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment. We introduce Decoupled Dynamic Neural Radiance Field (D$^2$NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background. Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes. A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting. To this end, we propose a novel loss to promote correct separation of phenomena. We further propose a shadow field network to detect and decouple dynamically moving shadows. We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.
1 Introduction
The paper frames monocular dynamic-scene reconstruction as an ill-posed 3D problem and introduces a self-supervised decomposition into dynamic, static, and shadow components. It targets separate synthesis, occluder and shadow removal, and dynamic-object segmentation.
- Monocular videos require disentangling camera motion from object motion while recovering a 3D static environment.
- Image-domain decomposition relies on 2D priors and inpainting, which lack 3D understanding and can produce sub-optimal results.
- Time-dependent scene modeling is severely ill-posed, while treating every scene part as time-dependent harms background reconstruction under limited network capacity.
- D2NeRF decouples dynamic and static components into separate radiance fields and uses a self-supervised skewed-entropy loss to encourage clean separation.
- A time-varying shadow field modulates the static component, capturing moving shadows while keeping its density and color static.
- The method supports separate free-view and time-varying synthesis, dynamic-occluder and shadow removal, and dynamic-object and shadow segmentation.
- The authors introduce synthetic and real-world datasets with rigid and non-rigid objects, rapid camera motion, and moving shadows, reporting better performance than state-of-the-art approaches.
2 Related Work
The related work situates D2NeRF among neural 3D scene representations, motion-decoupling methods, and image-level moving-object segmentation. Prior approaches vary in supervision, scene complexity, input requirements, and whether they synthesize isolated components.
- Scene Representations: A 3D scene representation encodes a scene’s geometry and appearance for downstream algorithms and applications.
- Scene Representations: NeRF uses differentiable volume rendering and multiple views to learn spatially and view-dependent radiance with an MLP.
- Motion Decoupling: Motion-decoupling methods include supervised systems using pretrained segmentation or motion masks and self-supervised systems with varying synthesis and decomposition capabilities.
- Motion Decoupling: The paper’s overview separates dynamic objects, static scenes, and non-static shadows into distinct radiance-field components.
- Motion Decoupling: STaR handles one rigid dynamic object with multi-view videos, whereas D2NeRF targets multiple non-rigid and topologically varying objects from monocular video.
- Image Segmentation of Moving Objects: Image-level moving-object segmentation commonly relies on motion clues such as differing optical-flow patterns.
3 Method
The method learns separate static and dynamic radiance fields from monocular video, adds self-supervised regularization for separation, and explicitly models moving shadows with a shadow field.
- 3.1 Composite Neural Radiance Field: Known camera poses and constant illumination support self-supervised reconstruction of separate static and dynamic scene components.The method targets freely moving monocular cameras and reconstructs a neural 3D representation.
- 3.1 Composite Neural Radiance Field: The static field models spatial density and view-dependent radiance, while the dynamic field additionally conditions on a per-frame time latent code.The two fields are composited through volumetric rendering along camera rays.
- 3.2 Supervision Losses: Photometric reconstruction alone permits the expressive dynamic field to absorb static occupancy, so self-supervised regularizers enforce decoupling.The factorization penalizes overlap between static and dynamic density by encouraging each spatial position to belong primarily to one component.
- 3.2 Supervision Losses: Skewed entropy with k > 1 favors static explanations, while excessive skewness can create fuzzy low-density floaters that ray regularization suppresses.The ray loss constrains the dynamic component to occupy few pixels and removes floaters on rays without dynamic objects.
- 3.2 Supervision Losses: Static regularization concentrates density along camera rays to reduce view-dependent artifacts caused by the camera pose and time being nearly one-to-one.This ambiguity arises because monocular casual videos rarely revisit exactly the same camera position.
- 3.3 Shadow Fields: A time-varying shadow field scales down static radiance, capturing shadows while keeping static density and color constrained.Explicit shadow modeling addresses ambiguity that can otherwise integrate frequent shadows into the static component or misrepresent them dynamically.
4 Experiments
The experiments evaluate D2NeRF on novel-view synthesis, video segmentation, and 3D scene decoupling using synthetic and real datasets. Ablations examine regularization and shadow-field behavior under challenging motion and shadow conditions.
- Evaluation: The evaluation covers novel-view synthesis, video segmentation, and 3D decoupling, with quantitative and qualitative results reported across three tasks.The main objective is removing dynamic objects and shadows while reconstructing the static environment.
- Datasets: The synthetic dataset contains five scenes with rigid or non-rigid dynamic objects, moving shadows, and ground-truth masks.Sequences use 200 frames generated from camera motion over randomly sampled azimuth and altitude keyframes.
- Datasets: The real-world dataset contains ten video sequences with rapid motion and non-trivial dynamic shadows, but lacks ground-truth views and masks for quantitative analysis.Additional data come from HyperNeRF and Nerfies settings.
- Scene decoupling: NSFF can fail by learning everything as dynamic, leaving an empty static component, because it lacks regularization specifically designed for scene decoupling.This comparison motivates the regularization used by D2NeRF.
- Ablations: The shadow-field network is necessary for decoupling large-area, slow or repetitive motion, and shadows highly correlated with the camera view.Qualitative shadow ablations are shown alongside quantitative and qualitative regularization ablations.
5 Conclusions
The paper concludes that D2NeRF decouples dynamic occluders and correlated shadows from casual monocular videos while recovering clean background representations and novel views. It also identifies camera registration, reflective surfaces, and ambiguous repetitive motion as limitations.
- Conclusions: D2NeRF performs self-supervised 3D scene decoupling and reconstruction from casual monocular videos.The method separately represents dynamic and static scene components.
- Conclusions: The skewed entropy regularizer separates dynamic from static components, while the shadow field removes dynamic shadows without explicitly modeling light–geometry interactions.The conclusion identifies both components as central to the method.
- Limitations: Accurate camera registration is required, while reflective surfaces and texture-less targets with small repetitive motion can cause incorrect or incomplete decoupling.The paper attributes the latter difficulty to ambiguous motion clues.
- Limitations: Alternative losses such as beta or Laplacian distributions may similarly separate components, potentially reducing hyperparameter complexity without affecting decoupling accuracy.The skewed entropy loss differs through its vanishing gradient near zero.
Checklist
The checklist records affirmative answers for claim-scope alignment, limitations, reproducibility materials, training details, asset licensing, and consent, while marking inapplicable ethics and theory items where appropriate.
- Claims and limitations: The paper states that its main claims accurately reflect its contributions and scope and that its limitations are described in Section 5.
- Ethics: Potential negative societal impacts and human-subjects requirements are marked not applicable, while the paper states that ethics guidelines were followed.
- Reproducibility: The authors report providing code, data, and reproduction instructions in supplementary material or through a URL, with training details in Section 4.2.They report no repeated evaluations because of computational-resource limitations.
- Theory: The checklist marks theoretical assumptions and complete proofs as not applicable.
- Assets and data: Existing assets are cited, licensed, and accompanied by information about new assets and consent for data involving people.Personally identifiable information is identified only for the cited “curls” dataset and is used under its license.
A Code and Data
The project page provides code, data, and additional video results, and static copies of the code and website are included in supplementary zip files.
- Release: Code, data, and additional video results are available on the D2NeRF project page.Static copies of the code and website are also included in the supplementary material as zip files.
B Hyperparameters
The evaluation uses scene-specific hyperparameter settings, with separate configurations for real-world and synthetic data. Training increases λs to promote separation, while the shadow-field setting varies by scene type.
- B Hyperparameters: The implementation uses a lightweight HyperNeRF dynamic component to reduce training time under limited TPU resources.It reduces samples per ray, batch size, and iterations relative to the original HyperNeRF settings, and omits background regularization requiring known background 3D points.
- B Hyperparameters: The evaluation uses five hyperparameter sets for real-world scenes and four for synthetic scenes.These settings were empirically established for varied scenes.
- B Hyperparameters: λs is increased during training to ensure that scenes separate fully into dynamic and static components.The schedule may increase linearly or exponentially, as indicated by the hyperparameter notation.
- B Hyperparameters: Table 3 separates configurations for mixed dynamic-object and shadow scenes, dynamic-shadow-only scenes, and synthetic scenes.Rows 1–4 cover mixed real-world scenes, row 5 dynamic shadows only, and rows 6–9 synthetic scenes.
- B Hyperparameters: Additional qualitative scene-decoupling results are provided for real-world and synthetic scenes in Figures 11–14.The figures include dynamic masks, decoupled components, static-background recovery, and novel-view comparisons with ground truth.
D Video Segmentation – Table 4, Figure 15
The paper evaluates image-level moving-object segmentation from the learned dynamic density, using synthetic scenes because existing benchmarks lack suitable frames or complete masks. Qualitative results compare the method with Motion Grouping and NeRF-based approaches.
- D Video Segmentation – Table 4, Figure 15: The learned dynamic density produces alpha masks that support image-level evaluation of moving-object segmentation.For NeRF-based methods, masks use normalized radiance weights from the time-varying component thresholded at 0.1.
- D Video Segmentation – Table 4, Figure 15: The synthetic dataset is used because existing video-segmentation benchmarks lack either sufficient frames for SfM or complete ground-truth masks.The stated benchmark limitations include too few frames and missing masks for dynamic objects and scene effects.
- D Video Segmentation – Table 4, Figure 15: Figures 11–14 additionally show masks, decoupled geometry, depth, and novel-view background recovery for real-world and synthetic scenes.Figure 14 compares unseen-view background renders against ground truth, with quantitative results in Table 1.
- D Video Segmentation – Table 4, Figure 15: Table 4 reports Jaccard index J and boundary measure F on training views.The method performs well on the “car” and “cars” scenes without skewed entropy because the background is clearly distinguishable from the moving object.
- D Video Segmentation – Table 4, Figure 15: Motion Grouping fails on the moving pillow, while NeRF-W produces severe cloud-like transient effects in the qualitative comparison.Motion Grouping segments everything except the table under its Hungarian setting.
- D Video Segmentation – Table 4, Figure 15: The proposed method achieves the best qualitative segmentation across the displayed scenes.The comparison is shown in Figure 15 alongside Motion Grouping and NeRF-W.
E Novel View Synthesis – Table 5, Figure 16
The paper evaluates whole-scene novel-view synthesis as a by-product of separately modeling static and time-varying targets. Comparisons include NeuralDiff and two HyperNeRF baselines under challenging view and motion conditions.
- E Novel View Synthesis – Table 5, Figure 16: Separate static and time-varying components enable evaluation of whole-scene novel-view synthesis after dynamic-scene reconstruction.The authors report that the static component fully utilizes network capacity and yields more robust reconstruction of dynamic objects and background.
- E Novel View Synthesis – Table 5, Figure 16: In rapidly moving or substantially different-view scenes, HyperNeRF fails to reconstruct a reasonable dynamic-object shape, while the proposed method can render high-fidelity views despite possible pose shifts.The examples include the “water” and “duck” scenes; the decoupled dynamic-object quality is slightly degraded relative to the decoupling result.
- E Novel View Synthesis – Table 5, Figure 16: The validation setup uses a camera as the validation view, creating greater challenges for all methods.These results are reported in Table 5 and Figure 16.
- E Novel View Synthesis – Table 5, Figure 16: Table 5 compares NeuralDiff, HyperNeRF base, and parameter-matched HyperNeRF fair against the proposed method.The three HyperNeRF scenes in the bottom row come from the HyperNeRF dataset.
F Ambiguity between Dynamic Component and Shadow
The shadow field models moving shadows as density-less radiance modulation while preserving static density and color. Shadow–object ambiguity remains a limitation, especially when object appearance resembles the background.
- F Ambiguity between Dynamic Component and Shadow: The shadow field represents density-less shadows more physically realistically and resolves ambiguity in their motion.Its purpose is to capture shadow effects while maintaining a static environmental representation.
- F Ambiguity between Dynamic Component and Shadow: For strong, fast-moving shadows, the dynamic component can learn shadows as thin layers over static geometry without the shadow field.Figure 17 illustrates this alternative representation, which tends to exclude unnecessary static-background texture.
- F Ambiguity between Dynamic Component and Shadow: Shadow–object ambiguity can cause dark or background-colored objects to be interpreted as moving shadows rather than 3D shapes.The resulting failed dynamic geometry reconstruction severely decreases novel-view synthesis performance, motivating a large λρ to suppress the shadow field in mixed scenes.