Source-linked AI summary
Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding
Boyu Cai, Li Yang, Yan Xu, Wei Liu, Nian Liu, Sikui Zhang, Yan Wang, Chunfeng Yuan, Weiming Hu
TL;DR
Dynamic inputs violate the static-scene assumptions of feed-forward NVS and open-vocabulary 3D understanding, causing spatial feature misalignment from transient objects. SPAR filters dynamic regions before latent aggregation and jointly trains motion estimation with photometric and semantic reconstruction. On D-RE10K, it achieves state-of-the-art dynamic NVS and motion-mask prediction, while semantic learning improves photometric reconstruction quality.
Problem
Static-scene assumptions make transient moving objects introduce conflicting geometry and semantics into shared 3D representations.
Method
SPAR uses joint semantic-geometric encoding, Cross-View Dynamic Region Prediction, and motion-mask-weighted photometric and semantic losses for self-supervised end-to-end training.
Results
SPAR achieves state-of-the-art performance on D-RE10K, including 22.15 dB and 23.33 dB PSNR with 3 and 4 views and 88.5% mIoU for motion-mask prediction.
Takeaways & Limitations
Semantic learning acts as a structural regularizer that improves photometric reconstruction quality while supporting robust, temporally stable 3D representations without explicit dynamic-object annotations.
Abstract
from arXiv · showhide
The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.
1 Introduction
SPAR addresses the failure of static-scene assumptions in dynamic, unposed environments by filtering transient regions before joint semantic-geometric encoding. Its dynamic-region-aware training supports state-of-the-art reconstruction and motion-mask prediction while revealing mutual benefits between photometric and semantic learning.
- Motivation: Static-scene assumptions cause moving objects to create conflicting geometry and semantics, producing RGB ghosting and inconsistent semantic predictions.Moving foregrounds occupy different 3D locations across frames and cannot be reconciled within one static representation.
- Proposed Framework: SPAR jointly synthesizes novel views and performs open-vocabulary semantic understanding from a few unposed observations of dynamic environments.Its encoder-decoder embeds RGB and semantic features into a latent scene representation and renders target-view images and semantic maps.
- Proposed Framework: The Cross-View Dynamic Region Predictor estimates motion masks and filters dynamic foregrounds before scene-level encoding to prevent transient content from contaminating the latent representation.This explicitly isolates moving regions before latent aggregation.
- Training: Predicted motion masks spatially weight photometric and semantic reconstruction losses, enabling end-to-end training without ground-truth motion-mask labels.The optimization focuses supervision on multi-view-consistent static regions while suppressing transient-foreground gradients.
- Results: SPAR achieves state-of-the-art performance on D-RE10K, reaching PSNR values of 22.15 dB with 3 views and 23.33 dB with 4 views.The model also reaches 88.5% mIoU for dynamic region prediction under 3 input views.
- Results: Ablations show that semantic learning acts as a structural regularizer that improves photometric reconstruction fidelity rather than compromising rendering capacity.The results support mutual benefits between photometric reconstruction and semantic scene understanding.
2 Related Work
Feed-forward 3D reconstruction and semantic understanding have advanced through generalized NVS, pose-free inference, large reconstruction models, token-space renderers, and language-aligned representations. SPAR extends this direction to dynamic environments by combining pose-conditioned tokens, dynamic-region masking, and joint rendering and understanding.
- Novel View Synthesis: Generalizable NVS predicts novel views or intermediate 3D representations in a single forward pass across diverse scenes.Later work improves sparse-view robustness and extends feed-forward reconstruction to efficient 3D Gaussian representations.
- Novel View Synthesis: Pose-free Gaussian inference reduces reliance on calibrated cameras, while RayZer introduces self-supervised pose-free NVS but still assumes static imagery.These approaches target efficient inference from unposed or sparsely observed inputs.
- Scene Understanding: Open-vocabulary scene understanding builds on vision-language pretraining, dense language-feature alignment, and query-based transformer segmentation.Representative systems include CLIP, LSeg, MaskFormer, and Mask2Former.
- Joint Reconstruction and Understanding: 3D semantic-rendering methods align language or semantic features with NeRFs or 3D Gaussians, but dense captures and per-scene optimization limit efficiency at scale.LSM addresses this drawback through 2D-to-3D feature alignment within a large feed-forward reconstruction model.
- SPAR: SPAR jointly processes dynamic multi-view images by constructing pose-conditioned photometric and semantic tokens, masking moving regions, and decoding target images and semantic features.Its overview describes a latent scene representation-based encoder-decoder architecture.
3 Method
SPAR jointly reconstructs photometric views and semantic features from dynamic, unposed multi-view inputs by masking transient regions before scene encoding and coupling motion estimation with reconstruction. Its shared latent representation and dynamic-region-aware supervision target multi-view-consistent scene understanding while reducing interference from moving foregrounds.
- Architecture: SPAR estimates camera poses, tokenizes photometric and semantic observations with pose rays, and processes them through a unified latent scene representation.The encoder aggregates masked multi-view tokens, while a shared decoder queries the representation for target-view outputs.
- Scene encoding: Masked photometric and semantic tokens are aggregated with learnable scene tokens through transformer self-attention to produce compact global scene tokens.The encoder enables cross-modal interaction among scene, photometric, and semantic tokens.
- Rendering: A joint rendering decoder separates target-view photometric and semantic features, predicting RGB values and continuous semantic embeddings with dual MLP heads.The decoder concatenates target-view query tokens with scene tokens, performs self-attention, and splits the outputs into photometric and semantic feature sets.
- Dynamic-region prediction: The Cross-View Dynamic Region Predictor uses cross-view inconsistencies to identify moving foreground regions before photometric and semantic tokens enter the encoder.It combines semantic, photometric, and ray features, then produces a probability map whose thresholded output forms the dynamic mask.
- Training: Dynamic-region-aware optimization weights rendering supervision toward static regions while reducing the influence of cross-view-inconsistent foreground motion.The predictor supplies spatial masks for photometric and semantic reconstruction, and the framework jointly optimizes rendering and dynamic-region estimation.
- Training: The unified loss uses pasted-object masks for binary-cross-entropy motion supervision alongside photometric, semantic, and regularization terms.Copy-paste augmentation supplies dynamic-object supervision without requiring ground-truth motion masks for real training sequences.
4 Experiments
Experiments evaluate SPAR for dynamic novel view synthesis, motion-region prediction, semantic reconstruction, cross-dataset generalization, and optimization components. Results show strong few-view performance, self-supervised motion estimation, qualitative robustness, and benefits from dynamic-region-aware and semantic-geometric training.
- Novel View Synthesis: 23.33 dB PSNR and 0.263 LPIPS are achieved with 4 input views, while 22.15 dB PSNR and 0.283 LPIPS are achieved with 3 views on D-RE10K.With 3 views, SPAR improves PSNR from 21.98 to 22.15 dB and reduces LPIPS from 0.314 to 0.283 compared with WildRayZer.
- Photometric-Semantic Reconstruction: Joint single-stage rendering reduces dynamic-scene rendering and segmentation errors relative to static-only and two-stage baselines.The two-stage WildRayZer + LSeg pipeline can propagate rendering artifacts into semantic predictions, whereas SPAR jointly fuses photometric and semantic features.
- Motion Mask Quality: 88.5% mIoU is achieved for motion-region prediction with 3 input views, outperforming the leading self-supervised baseline by a large margin.The predictor is trained without ground-truth motion-mask labels; SAM2 refinement is applied only at test time in the refined variant.
- Unbounded-Scene Generalization: SPAR predicts coherent dynamic masks and plausible novel views on unseen SpatialVID outdoor videos without training or fine-tuning on that dataset.The qualitative results include pedestrians, cyclists, occlusions, viewpoint changes, and unbounded layouts.
- Optimization Ablation: Dynamic-region-aware optimization provides larger robustness gains than random masking under severe dynamic interference.The targeted formulation supplies spatial guidance for separating dynamic interference and learning stable static scene representations.
- Ablation Studies: Adding the semantic branch consistently improves novel view synthesis across metrics, while removing pre-training, copy-paste augmentation, or SAM2 refinement degrades performance.Pre-training has the largest effect; removing copy-paste augmentation or test-time SAM2 refinement causes only minor degradation.
5 Conclusion
SPAR addresses static-scene assumptions in feed-forward 3D understanding by suppressing transient regions before scene aggregation and training motion estimation jointly with reconstruction. Evaluations report state-of-the-art dynamic NVS and motion-mask prediction, while semantic learning improves photometric reconstruction without explicit motion annotations.
- Conclusion: SPAR suppresses transient dynamic regions before latent scene aggregation through joint semantic-geometric encoding and a Cross-View Dynamic Region Predictor.Its self-supervised training scheme uses reconstruction loss and does not require ground-truth motion-mask labels.
- Conclusion: SPAR achieves state-of-the-art dynamic novel view synthesis and motion-mask prediction under extreme few-view settings on D-RE10K.The conclusion also reports strong inter-task synergy, with semantic learning acting as a structural regularizer for photometric reconstruction.