Source-linked AI summary
HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, Yiyi Liao
TL;DR
Urban scene understanding from RGB images requires joint reasoning about geometry, appearance, semantics, and motion, but prior approaches often focus on subsets or require costly 3D annotations. HUGS uses static and dynamic 3D Gaussians with physically constrained object motion to learn these elements jointly from noisy inputs. It supports real-time novel-view rendering, semantic outputs, and dynamic-scene reconstruction, while dynamic objects remain limited in their allowable rotation.
Problem
Urban scene understanding from RGB images must cover geometry, appearance, semantics, and moving objects, while existing methods often require additional inputs such as LiDAR or manually annotated 3D bounding boxes.
Method
HUGS jointly optimizes geometry, appearance, semantics, and motion with static and dynamic 3D Gaussians whose moving-object poses follow physical unicycle constraints.
Results
HUGS achieves state-of-the-art performance across novel-view appearance, semantic synthesis, and 3D semantic reconstruction while rendering new viewpoints in real time.
Takeaways & Limitations
The approach reconstructs dynamic urban scenes from sparse RGB images and noisy labels without requiring ground-truth 3D bounding boxes.
Takeaways & Limitations
Reconstructed dynamic objects can rotate only to a certain degree, and the model lacks control over additional degrees of freedom such as light editing.
Abstract
from arXiv · showhide
Holistic understanding of urban scenes based on RGB images is a challenging yet important problem. It encompasses understanding both the geometry and appearance to enable novel view synthesis, parsing semantic labels, and tracking moving objects. Despite considerable progress, existing approaches often focus on specific aspects of this task and require additional inputs such as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic urban scene understanding. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy, and reconstruct dynamic scenes, even in scenarios where 3D bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach.
1. Introduction
HUGS targets holistic urban scene understanding from posed RGB images, jointly modeling geometry, appearance, semantics, and motion. It decomposes scenes into static regions and rigidly moving objects, using physical motion constraints to handle noisy inputs and avoid ground-truth 3D bounding boxes.
- Urban scene understanding must support real-time novel-view synthesis, 2D and 3D semantics, depth, and dynamic-object tracking from posed RGB images.The motivation emphasizes holistic scene representation while minimizing sensor cost.
- Existing dynamic-scene methods often require costly ground-truth 3D bounding boxes, while per-frame pose optimization is sensitive to initialization and local minima.Prior methods also struggle with accurate 3D semantics and real-time rendering.
- HUGS uses 3D Gaussians to jointly infer geometry, appearance, semantics, and motion from predicted 2D semantic labels, optical flow, and 3D tracks despite their noise.The scene is decomposed into static regions and rigidly moving dynamic objects.
- Dynamic object poses are constrained by physical unicycle models, enabling decomposition from sparse urban images and noisy labels without ground-truth 3D bounding boxes.The approach extends Gaussian Splatting with semantic, flow, camera-exposure, and dynamic-object modeling.
- HUGS achieves state-of-the-art performance across novel-view appearance, semantic synthesis, and 3D semantic reconstruction benchmarks.
2. Related Work
Related work spans 2D scene understanding, neural rendering, urban reconstruction, and dynamic-scene modeling. These approaches improve individual capabilities but commonly face limitations in 3D consistency, appearance modeling, or required dynamic-object annotations.
- 3D Scene Understanding: 2D methods predict semantics, depth, and optical flow from images but can lack profound 3D understanding and multi-view consistency.
- 3D Scene Understanding: Neural rendering lifts 2D information into 3D space, with NeRF-based methods improving scene-understanding quality and efficiency.
- Urban Scene Reconstruction: Point-based and mesh-based urban reconstruction can be faithful but struggle with complete high-quality appearance modeling, whereas NeRF-based methods support high-quality novel viewpoints.
- Urban Scene Reconstruction: Dynamic urban-scene methods commonly rely on accurate 3D bounding boxes to separate moving objects from static components.PNF instead uses monocular 3D box predictions and jointly optimizes object poses.
- Gaussian Splatting: HUGS decomposes scenes into static regions and rigidly moving dynamic objects, representing each dynamic object with canonical-space 3D Gaussians transformed by unicycle-constrained motion.The same Gaussians encode appearance and semantics, while Gaussian-center motion provides optical flow.
- Gaussian Splatting: Dynamic Gaussian Splatting methods may require many training views and ground-truth masks, while other approaches decompose static and dynamic components without resolving these broader requirements.
3. Method
HUGS represents urban scenes with static and dynamic 3D Gaussians that jointly encode appearance, semantics, geometry, and motion. It renders multiple modalities and uses noisy RGB-derived predictions, physical motion constraints, and multimodal losses for efficient reconstruction and inference.
- Scene representation: HUGS decomposes urban scenes into static and dynamic 3D Gaussians, with dynamic vehicles represented in canonical coordinates and transformed over time.The representation includes Gaussian position, covariance, opacity, appearance, semantic logits, and motion-related information.
- Dynamic motion: The unicycle model parameterizes vehicle states with position and yaw, while forward and angular velocities impose physically constrained motion.The authors use trainable states with regularization because direct recursive parameterization is difficult to optimize.
- Multimodal rendering: The representation renders RGB images, semantic labels, and optical flow through volume rendering from the Gaussian attributes.Optical flow is obtained by projecting Gaussian centers at two timestamps, while semantic logits are rendered into 2D labels.
- Semantic reconstruction: Applying softmax to 3D semantic logits before alpha blending reduces floaters compared with normalizing accumulated 2D semantic logits.The 3D normalization prevents a single high-logit sample from masking an undesired floating point along a ray.
4. Experiments
HUGS is evaluated across novel-view synthesis, semantic reconstruction, and downstream scene understanding on KITTI, vKITTI, and KITTI-360. It consistently handles noisy dynamic-object tracks, achieves strong static-scene performance, and supports decomposition and editing.
- Experimental Setup: Experiments span novel-view synthesis, novel semantic synthesis, and 3D semantic reconstruction on KITTI, Virtual KITTI 2, and KITTI-360.The evaluation uses predicted or noisy 3D tracking inputs for dynamic scenes and compares against established dynamic, appearance, and semantic baselines.
- 4.1. Novel View Synthesis: HUGS consistently outperforms baselines for dynamic-scene novel-view synthesis with predicted or noisy 3D trackings.Compared with NSG and MARS, HUGS reconstructs dynamic objects with sharper details, including under more severe noise in vKITTI.
- 4.1. Novel View Synthesis: HUGS achieves state-of-the-art static-scene performance on the KITTI-360 leaderboard and comparable novel-view semantic synthesis performance to PNF.The static leaderboard contains five sequences; the semantic comparison does not use PNF’s category-level prior.
- 4.2. Semantic and Geometric Scene Understanding: The Gaussian representation produces a more accurate 3D semantic point cloud than Semantic Nerfacto, although Semantic Nerfacto can render faithful 2D semantic labels.Evaluation uses KITTI-360 LiDAR points, with all Table 3 metrics calculated in 3D space.
- 4.3. Downstream Applications: The decomposed representation supports separating foreground moving objects from background and editing scenes by swapping, rotating, or translating dynamic objects.These capabilities are demonstrated qualitatively on KITTI.
- 4.4. Ablation Study: Ablations show that the unicycle model improves rendering quality and 3D tracking accuracy under noisy boxes, while exposure modeling matters for scenes with strong exposure variance.Flow supervision contributes to underlying geometry through explicit pixel correspondences, whereas semantic and flow losses contribute little to appearance quality.
5. Conclusion
HUGS jointly optimizes appearance, geometry, and motion for urban scenes, achieving state-of-the-art performance across various tasks. The method remains limited in dynamic-object rotation and broader control capabilities.
- HUGS jointly optimizes appearance, geometry, and motion as a holistic representation for urban scenes.
- The method achieves state-of-the-art performance on various tasks.
- Reconstructed dynamic objects can only rotate to a certain degree.The authors suggest category-level priors as a future direction for accurate full-object reconstruction.
- HUGS lacks control over additional degrees of freedom, such as light editing.The paper identifies expanded control based on the Gaussian representation as a promising direction.
A. Implementation
The implementation represents scenes with projected 3D Gaussians and uses shared volume rendering to produce appearance, semantics, depth, and flow. CUDA execution preserves real-time rendering across modalities.
- Each Gaussian uses rotation, scale, opacity, and spherical harmonics to define its 3D covariance and appearance.The covariance is transformed into camera coordinates and reduced to a 2D covariance for splatting.
- EWA splatting reduces the camera-coordinate covariance to a 2 × 2 matrix by skipping its third row and column.
- Volume rendering accumulates Gaussian color and also renders semantic features, depth, and optical flow.The same rendering technique is applied across these modalities using per-Gaussian attributes and motion.
- CUDA computes projected opacity and Gaussian sorting once for all modalities, maintaining real-time rendering.
- The default semantic rendering uses 3D softmax, which normalizes logits for each 3D point rather than after 2D accumulation.The paper contrasts this with 2D softmax used in most NeRF-based semantic reconstruction methods.
A.3. Metrics
The paper evaluates appearance, semantic synthesis, 3D semantic reconstruction, tracking, and depth using task-specific image, geometric, semantic, pose, and depth metrics.
- Novel view appearance synthesis is evaluated with PSNR, SSIM, and LPIPS.
- Novel view semantic synthesis is evaluated using mean Intersection over Union (mIoU).
- 3D semantic reconstruction compares extracted semantic point clouds with ground-truth LiDAR points using geometric and semantic metrics.Geometric quality includes accuracy and completeness, measuring distances between reconstructed and LiDAR points.
- 3D tracking evaluates predicted object poses against ground-truth poses using rotation and translation accuracy metrics.The metrics are denoted eR for rotation accuracy and et for translation accuracy.
- Depth estimation measures L2 distance between projected LiDAR depth and predicted depth on pixels with valid LiDAR projections.
B. Data
Experiments use KITTI, Virtual KITTI 2, and KITTI-360, with comparisons against established urban-scene baselines. The evaluation includes dynamic scenes with noisy tracking inputs and reports time consumption separately.
- Experiments use KITTI, Virtual KITTI 2, and KITTI-360 across static and dynamic urban-scene settings.KITTI and Virtual KITTI 2 use selected frames from Scenes 02 and 06, while KITTI-360 includes official leaderboard sequences and dynamic sequence 00 frames.
- Table 6 reports the time-consumption breakdown of the method.
- The baselines include NSG, MARS, PNF, and Semantic Nerfacto.
- NSG decomposes dynamic scenes into static background and dynamic foreground components and provides a stable implementation used here.
- The updated MARS implementation improves PSNR by 3 to 4 dB over the original paper’s reported numbers.
- PNF is compared on KITTI-360 leaderboard tasks because it optimizes noisy 3D bounding boxes of dynamic objects.
- Semantic Nerfacto serves as a comparison for 3D semantic point-cloud geometry and combines several real-data techniques with a semantic head.
D. Additional Experiment Results
The runtime breakdown isolates preparation and rendering components while showing that unicycle modeling, dynamic decomposition, and depth rendering add little time.
- Preparation includes tile partitioning and Gaussian sorting, while π denotes volume rendering and affine denotes affine transformation.
- The unicycle model, dynamic decomposition, and depth rendering are excluded from the breakdown because they consume hardly any additional time.
D.2. Additional Comparison Experiments
Additional comparisons show strong performance across dynamic-scene settings, tracking, semantic reconstruction, and initialization, while Semantic Nerfacto preserves meaningful semantics but struggles with geometry.
- The approach outperforms NSG and MARS across all test scenes when evaluated with ground-truth 3D trackings.
- Both LiDAR- and COLMAP-based initialization outperform random initialization, with COLMAP showing a slight advantage over LiDAR.
- Semantic Nerfacto renders meaningful novel-view semantic images but struggles to accurately reconstruct correct geometry.
- The method consistently improves QD-3DT across two KITTI scenes after initializing its optimized pose with QD-3DT.
D.3. Additional Ablation Experiments
Ablations show that 3D semantic normalization, optical-flow supervision, semantic supervision, and informed initialization improve reconstruction quality under the tested settings.
- 3D semantic-logit normalization yields notable qualitative and quantitative improvement compared with 2D normalization.
- Both semantic loss and optical-flow loss improve underlying geometry, with optical flow producing the clearer enhancement.Optical flow improves geometry by guiding correspondences across neighboring frames.
- LiDAR- and COLMAP-based initialization outperform random initialization, while COLMAP slightly surpasses LiDAR in the comparison.The paper adopts COLMAP-based initialization by default.
- COLMAP improves far-distance object quality, whereas unobserved LiDAR points can create artifacts in test viewpoints.
D.4. Visualization of Optimization Progress
Visualization results show joint optimization of static background and dynamic-object trajectories, with physical constraints smoothing noisy motion estimates and supervision improving rendered geometry and flow.
- Physical constraints from the unicycle model produce a smooth bounding-box trajectory from noisy 3D bounding boxes.
- Semantic or optical-flow loss improves geometry, while optical-flow loss provides the more evident improvement.
- Semantic supervision mitigates artifacts in rendered optical flow, and pseudo-optical-flow supervision contributes further improvement.
- LiDAR- and COLMAP-based initialization are superior to random initialization, although LiDAR can introduce artifacts near the ego car.
- The method jointly optimizes the static background and dynamic foreground-object trajectories.