Source-linked AI summary
SUDS: Scalable Urban Dynamic Scenes
Haithem Turki, Jason Y. Zhang, Francesco Ferroni, Deva Ramanan
TL;DR
SUDS targets dynamic city-scale reconstruction, where existing methods struggle with multi-video scenes and often require object-level annotations. It combines a three-branch hash-table representation with unlabeled RGB, LiDAR, feature, and optical-flow signals. The resulting system scales to large dynamic environments, supports multiple downstream tasks, and surpasses state-of-the-art methods in reported benchmarks, while retaining important boundary and flow-supervision limitations.
Problem
Dynamic urban reconstruction must jointly model fixed, transient, and moving geometry across limited-viewpoint videos, while prior methods often require difficult-to-acquire labels.
Method
SUDS uses static, dynamic, and far-field branches backed by multiresolution hash tables, trained with unlabeled RGB, LiDAR, self-supervised features, and optical flow.
Results
SUDS scales dynamic reconstruction across thousands of videos and supports scene flow, free-viewpoint synthesis, and unsupervised scene understanding while surpassing state-of-the-art reconstruction methods.
Takeaways & Limitations
The representation provides a modular route to large dynamic environments with semantic decomposition and motion-aware rendering from unlabeled inputs.
Takeaways & Limitations
DINO-based flow correspondences are sparse and expensive, and dynamic objects remain video-specific, preventing straightforward motion extrapolation beyond captured video boundaries.
Abstract
from arXiv · showhide
We extend neural radiance fields (NeRFs) to dynamic large-scale urban scenes. Prior work tends to reconstruct single video clips of short durations (up to 10 seconds). Two reasons are that such methods (a) tend to scale linearly with the number of moving objects and input videos because a separate model is built for each and (b) tend to require supervision via 3D bounding boxes and panoptic labels, obtained manually or via category-specific models. As a step towards truly open-world reconstructions of dynamic cities, we introduce two key innovations: (a) we factorize the scene into three separate hash table data structures to efficiently encode static, dynamic, and far-field radiance fields, and (b) we make use of unlabeled target signals consisting of RGB images, sparse LiDAR, off-the-shelf self-supervised 2D descriptors, and most importantly, 2D optical flow. Operationalizing such inputs via photometric, geometric, and feature-metric reconstruction losses enables SUDS to decompose dynamic scenes into the static background, individual objects, and their motions. When combined with our multi-branch table representation, such reconstructions can be scaled to tens of thousands of objects across 1.2 million frames from 1700 videos spanning geospatial footprints of hundreds of kilometers, (to our knowledge) the largest dynamic NeRF built to date. We present qualitative initial results on a variety of tasks enabled by our representations, including novel-view synthesis of dynamic urban scenes, unsupervised 3D instance segmentation, and unsupervised 3D cuboid detection. To compare to prior work, we also evaluate on KITTI and Virtual KITTI 2, surpassing state-of-the-art methods that rely on ground truth 3D bounding box annotations while being 10x quicker to train.
1. Introduction
SUDS addresses the difficulty of reconstructing large, dynamic urban environments across multiple videos. It combines unlabeled sensor and motion signals with a factorized representation that scales city-wide and supports downstream scene understanding.
- Dynamic urban maps remain limited because static representations cannot capture moving vehicles, pedestrians, and other objects.
- Reconstructing multiple videos must distinguish globally fixed geometry, transient objects, and truly dynamic objects under limited-viewpoint observations.
- SUDS uses LiDAR depth and optical flow instead of requiring difficult-to-acquire panoptic labels or bounding boxes.
- Its three components model stationary topography, transient and moving objects, and far-field objects or sky using partitioned multiresolution hash tables.
- SUDS introduces a large-scale dynamic NeRF and demonstrates free-viewpoint synthesis, 3D scene flow, unsupervised instance segmentation, and 3D cuboid detection.
2. Related Work
Related work establishes the challenges of scaling NeRFs, modeling dynamics, and adding semantics without prohibitive memory or annotation requirements. SUDS builds on spatial partitioning, optical-flow supervision, feature hashing, and shadow-aware decomposition while targeting multi-video urban scenes.
- Scale: Spatial partitioning and multiresolution methods address unbounded or large scenes, but fixed-size MLPs become blurry, unstable, or computationally intractable at scale.
- Dynamics: Dynamic NeRFs use time conditioning, deformation fields, optical flow, or canonical templates, while many approaches remain focused on narrower settings.
- Object-centric approaches: Object-centric methods compose per-object models with a background, but memory grows with the number of objects and some cannot handle ego-motion or multi-video representations.
- Semantics: Semantic and panoptic NeRF methods add semantic outputs but rely heavily on accurate annotations that are difficult to obtain in the wild at scale.
- Fast training and depth: SUDS uses hash tables as the base of a three-branch representation and applies LiDAR supervision to dynamic environments.
3. Approach
SUDS represents dynamic urban scenes with static, dynamic, and far-field branches backed by multiresolution hash tables and spatial partitioning. It trains these branches with photometric, feature, depth, flow, and factorization objectives using unlabeled signals.
- Representation: SUDS factorizes each scene into static topography, video-specific dynamic objects, and a far-field environment map for distant objects and sky.
- Representation: Each branch uses multiresolution hash tables followed by a small MLP, avoiding the prohibitive cost of conventional MLP-only NeRFs at target scales.
- Hash tables: Static branches use 3D spatial voxels, dynamic branches use 4D spacetime voxels, and far-field branches use direction-indexed environment-map voxels.
- Hash tables: Video identifiers index distinct dynamic functions without being used for interpolation, preventing unnatural averaging across different movers.
- Appearance and semantics: An explicit shadow field scales static color pointwise, while feature distillation from DINO supplies semantic awareness and scene flow predicts temporal motion consistency.
- Optimization: Spatial cells are trained as separate models, and optimization combines reconstruction, warping, flow, factorization, feature, and LiDAR depth losses.
4. Experiments
SUDS is evaluated on city-scale reconstruction, downstream geometric and semantic tasks, and benchmark novel-view and image reconstruction. It consistently outperforms prior methods, while flow-based warping is the most important ablated input.
- Experimental setup: SUDS is evaluated on City-1M, KITTI, and Virtual KITTI 2 using view synthesis, image reconstruction, instance segmentation, and cuboid detection.The experiments also include semantic segmentation and component ablations.
- City-Scale Reconstruction: SUDS outperforms all Mega-NeRF variants by a large margin on City-1M city-scale view synthesis.The comparison uses 48 spatial cells and includes Mega-NeRF, Mega-NeRF-T, and Mega-NeRF-A.
- City-Scale Reconstruction: Instance segmentation and 3D cuboid detection require no additional labels because both use geometric clustering of the learned representation.Cuboids are derived from PCA of the convex hull of points assigned to each 3D instance.
- KITTI Benchmarks: SUDS surpasses state-of-the-art methods in KITTI image-reconstruction PSNR and SSIM under the prior experimental setup.The comparison relies on reported PNF numbers because its implementation is unavailable.
- KITTI Benchmarks: SUDS achieves the best novel-view-synthesis results across all KITTI and Virtual KITTI 2 training splits and metrics.Performance declines for every method as training views decrease, while SUDS remains consistently ahead.
- Diagnostics: Flow-based warping is the single most important input in the ablation, whereas depth is the least crucial input.The result suggests SUDS can generalize to settings without depth measurements.
5. Conclusion
SUDS provides a modular, large-scale representation for dynamic neural scenes by encoding static geometry and transient objects across thousands of videos. It uses unlabeled inputs to support semantic awareness and scene flow, while acknowledging that photorealistic city-scale environments remain an open challenge.
- Conclusion: SUDS uses a multi-branch hash table structure to disentangle and efficiently encode static geometry and transient objects across thousands of videos.The conclusion describes this as a modular approach to dynamic neural representations at previously unexplored scale.
- Conclusion: Unlabeled inputs enable SUDS to learn semantic awareness and scene flow and perform downstream tasks while surpassing methods that rely on human labeling.The stated downstream capability is within the scope of the learned dynamic representation.
- Conclusion: Building truly photorealistic city-scale dynamic environments remains an open challenge.The conclusion characterizes SUDS as a first attempt rather than a complete solution.
A. Tracking
SUDS uses feature-based correspondences to track masks and keypoints across frames, including through 2D occluders, after instance detection.
- SUDS computes mask- and keypoint-level correspondences across frames using Best-Buddies similarity on features within or between instances.
- As a 3D representation, SUDS can track correspondences through 2D occluders.
- The proposal network samples points along each camera ray using separate static and dynamic hash-table-backed branches.
- SUDS regularizes scene flow with spatial and temporal smoothness priors adopted from NSFF.
D. Ablation Details
The ablations remove reconstruction, warping, flow, factorization, appearance, and occlusion components to isolate their contributions to SUDS.
- The ablations remove depth, optical-flow, and warping-related terms from the reconstruction objective.
- Additional ablations remove the appearance embedding and occlusion weights used to downweight warping losses.
- The study separately removes static-dynamic factorization by using a single time-dependent branch.
- The loss components are grouped into reconstruction, warping, and flow losses for ablation.
E. Additional Training Details
City-1M is partitioned into geographically clustered cells for training, with each cell trained independently and all cells fitting on one GPU at inference.
- City-1M is divided into 48 camera-based k-means cells, each covering 2.9 km2 and 32k frames across 98 videos on average.
- Each cell is trained on one A100 GPU for 2 days, with the same training setup used for each KITTI scene.
- All cells can be fit on a single A100 GPU at inference time.
F. Assets
The paper describes City-1M’s synchronized camera-LiDAR collection, reports scaling behavior across coverage and data volume, and identifies the software and dataset assets used.
- City-1M uses seven ring cameras capturing 2048x1550 images at 20 Hz with a combined 360° field of view.
- Synchronized VLP-32C LiDAR sensors produce point clouds averaging 100,000 points at 10 Hz, while camera poses combine GPS- and sensor-based localization.
- City-1M cell quality degrades sublinearly across metrics, with image and video counts having the largest impact.
- SUDS primarily uses Nerfstudio, tiny-cuda-nn, OpenCV, Scikit, and a feature extractor implementation, while KITTI and VKITTI2 provide the listed datasets.
G. Limitations
SUDS has practical limitations involving resource demands, input quality, scene boundaries, shadow attribution, instance-level performance, and privacy. These constraints limit extrapolation, reconstruction quality, scalability, and deployment safety.
- Dynamic objects are video-specific, so SUDS cannot extrapolate their motion beyond captured video boundaries or straightforwardly render across disjoint sequences.
- High-quality optical flow affects model performance, while completely incorrect flow degrades quality; SUDS also assumes linear flow between observed timestamps.
- More than 20TB of compressed intermediate storage is required for city-scale preprocessing and training data preparation.This includes extracting DINO features, computing optical flow, deriving coordinate bounds, and storing randomized training batches.
- Persistent shadows can be misattributed to static topology when they appear in all observations of a location.This can occur even when the shadow is correctly assigned to the dynamic branch.
- SUDS provides initial qualitative instance-level results but is not competitive with conventional approaches.
- The model retains captured privacy-sensitive information such as faces and license plates even when semantic filtering removes categories or components at render time.The paper identifies preprocessing as a possible mitigation.