Source-linked AI summary
OmniRe: Omni Urban Scene Reconstruction
Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, Li Song, Yue Wang
TL;DR
Existing dynamic-scene methods do not holistically reconstruct the diverse rigid and non-rigid actors needed for urban digital-twin simulation. OmniRe combines 3D Gaussian scene graphs with actor-specific canonical representations and reports strong reconstruction results, simulation capabilities, and generalization across driving datasets.
Problem
Existing methods primarily focus on vehicles, while comprehensive urban simulation requires reconstruction of diverse actors including pedestrians and cyclists.
Method
OmniRe uses a dynamic neural scene graph based on 3D Gaussian Splatting, with specialized representations for rigid and non-rigid actors and per-node deformation fields.
Results
OmniRe outperforms other methods in appearance, geometry, and novel-view reconstruction, with especially large gains in human-related regions.
Takeaways & Limitations
The representation supports controllable scene editing and simulations involving pedestrians, human behavior, and human-vehicle interaction.
Takeaways & Limitations
OmniRe does not explicitly model lighting and produces less satisfactory novel views when cameras deviate substantially from training trajectories.
Abstract
from arXiv · showhide
We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hindering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-length reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. OmniRe allows holistically reconstructing any dynamic object in the scene, enabling advanced simulations (~60Hz) that include human-participated scenarios, such as pedestrian behavior simulation and human-vehicle interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We further extend our results to 5 additional popular driving datasets to demonstrate its generalizability on common urban scenes.
1 INTRODUCTION
OmniRe addresses the challenge of reconstructing diverse, moving urban actors by combining Gaussian scene graphs with actor-specific representations. It targets controllable, human-centered digital-twin simulations beyond vehicle-focused methods.
- Motivation: Photorealistic 4D digital twins support high-fidelity simulation, algorithm training, and closed-loop evaluation of autonomous-driving systems.Data-driven neural representations offer an alternative to artist-generated assets as scale, diversity, and realism demands increase.
- Limitations of prior work: Prior approaches either reconstruct static and time-dependent fields or use scene graphs, but scene-graph methods mainly model rigid vehicles and neglect pedestrians and cyclists.The former lack editability and direct control by classical behavior models.
- Approach: OmniRe models vehicles, pedestrians, cyclists, and other dynamic actors for interactive simulation, including precise joint-level human control.The framework addresses unified modeling of diverse non-rigid actors and human behavior relevant to driving safety.
- Approach: OmniRe builds a 3D Gaussian Splatting dynamic neural scene graph with dedicated Gaussian representations for dynamic actors in local canonical spaces.The framework combines static backgrounds, vehicles, and non-rigid actors in one reconstruction system.
- Contributions: The paper reports holistic reconstruction, human-centered simulations, generalization across six driving datasets, and state-of-the-art scene reconstruction and novel view synthesis.
2 RELATED WORK
Related work develops neural and Gaussian representations for dynamic scenes, while human-specific methods use body models to represent and control non-rigid motion. OmniRe builds on these directions with Gaussian scene graphs spanning rigid and non-rigid actors.
- Dynamic Scene Modeling: Dynamic scene reconstruction includes deformation-based neural representations and newer Gaussian Splatting methods.
- Dynamic Scene Modeling: Gaussian scene graphs combine representations for rigid and non-rigid objects to improve flexibility and controllability across diverse actors.
- Human Modeling: Human-modeling methods use SMPL with Gaussian representations or canonical-space warping to reconstruct non-rigid bodies and enable fine control.
3 PRELIMINARIES
The preliminaries describe 3D Gaussian Splatting as a scene representation rendered by depth-sorted alpha blending, together with rigid transforms and SMPL-based human deformation.
- 3D Gaussian Splatting: 3D Gaussian Splatting represents a scene as colored Gaussian blobs with opacity, position, rotation, anisotropic scale, and view-dependent color attributes.
- 3D Gaussian Splatting: Pixel colors are computed by sorting overlapping Gaussians by camera distance and alpha-blending their projected contributions.
- Rigid Transformation: A rigid transformation applies rotation and translation to Gaussian positions while rotating their orientations.
- SMPL Model: SMPL shapes and poses a canonical human template using shape and pose parameters, then deforms vertices through joint transformations and linear blending skinning.
- SMPL Model: OmniRe obtains pedestrian pose sequences and shape parameters, updates joint transformations over time, and interpolates them onto Gaussians for non-rigid deformation.
4 METHOD
OmniRe reconstructs urban scenes as controllable Gaussian scene graphs that combine static backgrounds, rigid vehicles, and diverse non-rigid actors. Specialized human-pose processing and per-node deformation support faithful reconstructions under occlusion and complex motion.
- Optimization: End-to-end optimization combines image, depth, opacity, and representation-specific regularization losses to obtain faithful and controllable reconstructions.Rendered depth is supervised with sparse LiDAR depth, while opacity is encouraged to align with the non-sky mask.
- Gaussian Scene Graph: OmniRe represents urban scenes with Gaussian scene graphs containing sky, static background, rigid-object, and non-rigid-object nodes.Foreground Gaussians are defined in local or canonical spaces, transformed into world space, rasterized for images and depth, and jointly optimized with reconstruction losses.
- Rigid Nodes: Rigid vehicle Gaussians remain fixed in local space while their world-space positions change through time-dependent pose transformations.This representation models vehicles as rigidly movable objects using poses in SE(3).
- Non-Rigid Nodes: Non-rigid nodes separately address global motion and local deformation for pedestrians, cyclists, and other dynamically deforming individuals.SMPL nodes drive human Gaussians with pose and shape parameters, while deformable nodes cover template-less or out-of-distribution non-rigid actors.
- Non-Rigid Nodes: A shared deformation network predicts per-node changes in position, rotation, and scale, with instance embeddings distinguishing nodes while retaining computational efficiency.The network applies changes to canonical Gaussians; the method reports that deformable Gaussians are essential for reconstruction quality in complex outdoor scenes.
- In-the-Wild Human Reconstruction: The human-pose pipeline matches identities across camera views and completes poses missing because of occlusions in driving logs.It processes camera videos independently, matches predicted tracklets using projected IoU, and interpolates missing poses to produce temporally consistent human poses.
5 EXPERIMENTS
OmniRe is evaluated on Waymo using appearance, geometry, qualitative reconstruction, and component ablations. It outperforms prior methods while reconstructing diverse dynamic actors and supporting controllable simulation.
- Appearance: On Waymo, OmniRe is evaluated for scene reconstruction and novel view synthesis using PSNR and SSIM across full-image, human-related, and vehicle-related regions.Every 10th frame is held out for novel view synthesis.
- Appearance: OmniRe outperforms all compared methods by a significant margin in human-related regions and is slightly better than StreetGS in vehicle regions.The authors attribute the vehicle-region difference to StreetGS lacking human modeling, allowing human-region supervision to affect vehicle modeling.
- Appearance: Qualitatively, OmniRe reconstructs fine details for vehicles, pedestrians, and cyclists while handling occlusion, deformation, and extreme motion.Compared methods exhibit blurry novel views, unrecovered pedestrians, motion blur, or coarse moving-object structures.
- Geometry: OmniRe outperforms other methods by a large margin on LiDAR-depth RMSE and two-way Chamfer Distance for training and novel frames.Rendered LiDAR visualizations show accurate reconstruction of dynamic humans and vehicles.
- Ablation Studies: Ablations show that SMPL modeling improves leg reconstruction and joint-level control, while pose refinement addresses noisy pose errors and deformable nodes improve template-less actor reconstruction.Bounding-box refinement further improves numeric metrics and reduces foreground blurriness.
- Applications to Simulation: Because instances are modeled separately, OmniRe supports asset position and rotation editing, cross-scene asset transfer, and reenacted pedestrian–vehicle interactions.The paper demonstrates vehicle swaps, pedestrian insertion, and a vehicle stopping while a pedestrian crosses.
6 CONCLUSION
OmniRe provides comprehensive urban scene modeling with Gaussian Scene Graphs for driving and robotics simulation. The paper also identifies lighting and substantially displaced cameras as remaining limitations.
- OmniRe tackles comprehensive urban scene modeling using Gaussian Scene Graphs and achieves fast, high-quality reconstruction and rendering.
- The method aims to support autonomous-driving simulation that can enhance development and testing while safety and controllability remain open challenges.
- OmniRe does not explicitly model lighting, which can cause visual-harmony issues when simulations combine elements reconstructed under different lighting conditions.
- Novel views become less satisfactory when the camera deviates significantly from the training trajectories, similarly to other per-scene optimization methods.
7 ETHICS STATEMENT
The work uses established public datasets rather than collecting or annotating new data, with privacy protections for identifiable human features.
- OmniRe uses established public datasets and does not collect or annotate new data.
- The datasets blur or anonymize sensitive information, including identifiable human features, to protect individual privacy.
- The authors emphasize responsible and ethical use of the method and future applications to maintain safety and preserve privacy.
A IMPLEMENTATION DETAILS
The implementation initializes Gaussian representations from LiDAR, random samples, bounding boxes, and template meshes, then jointly optimizes scene nodes with reconstruction and regularization losses. Evaluation comparisons use consistent camera and LiDAR-supervision settings across methods.
- Initialization: The background initialization combines 6 × 10^5 LiDAR points with 4 × 10^5 random samples, split into near and far samples.
- Initialization: Rigid and deformable nodes are initialized from accumulated LiDAR points within bounding boxes, while SMPL nodes use canonical template meshes.
- Training: All scene nodes are optimized jointly for 30,000 iterations, with node-specific Gaussian rotation learning rates.
- Optimization: Dynamic regions receive an image-loss weight of 5 because sparse temporal-spatial observations provide insufficient supervision.
- Optimization: The method adds regularization terms for different Gaussian representations, including pose smoothness for human body poses.
C EVALUATION
OmniRe is evaluated through image-quality, dynamic-region, and geometry metrics on Waymo scenes, including crowded, nighttime, adverse-weather, and high-speed conditions. The reported results generally favor OmniRe, while high-speed scenes show comparable performance with StreetGS.
- Appearance: Novel View Synthesis uses PSNR and SSIM for rendered images, including vehicle, human, and movable dynamic regions identified with semantic and bounding-box masks.
- Geometry: Geometry evaluation compares rendered depth with sparse LiDAR depth and includes Chamfer Distance after matching depth pixels to LiDAR rays.
- Challenging scenes: OmniRe maintains high reconstruction quality across various challenging scenes in comparisons with StreetGS, PVG, and DeformableGS.
- Super Crowded Scenes: OmniRe performs well in three extremely crowded Waymo scenes containing large groups of people crossing streets.
- Nighttime Scenes: OmniRe outperforms compared methods on three nighttime scenes, achieving superior reconstruction quality under low-light conditions.
- Adverse Weather Conditions: OmniRe maintains high reconstruction fidelity across seven rainy, foggy, and cloudy scenes.
- High-Speed Scenes: On high-speed scenes lacking non-rigid objects, OmniRe and StreetGS demonstrate comparable performance, so human metrics are not applicable.
D.4 ABLATION STUDIES
The ablations examine AbsGrad and report that it has only a marginal effect on performance, while the evaluation uses dynamic Waymo scenes and expanded result tables.
- AbsGrad Ablation: Disabling AbsGrad decreases performance by about 0.1 PSNR across methods, indicating it does not explain OmniRe’s performance advantage.DeformableGS also fails to run without AbsGrad because of out-of-memory issues.
- Evaluation Setup: The Waymo evaluation includes eight dynamic scenes used for testing in the reported ablations.The scenes contain diverse dynamic classes, including pedestrians and cyclists, and each segment contains approximately 150 frames.
- Additional Ablations: The supplementary tables expand earlier results with SSIM evaluations and report additional ablations on ground-truth box refinement and non-rigid modeling.Tables 13 and 14 provide full versions of selected evaluations and corresponding ablation studies.
E OM N IRE IN PRACTICE
OmniRe uses controllable scene representations for diverse urban actors, with distinct handling of pedestrians and dataset-dependent labeling limitations. Its assets can also be flexibly edited.
- Controllability: Bounding boxes provide controllable manipulation of rigid vehicles and individual human body movements for scene simulation.The paper notes that common autonomous-driving datasets generally provide these annotations through established auto-labeling tools.
- Asset Editing: OmniRe supports flexible editing of scene assets.The paper presents this capability as a figure-level example of practical scene manipulation.
- Pedestrian Modeling: Near-range pedestrians use SMPL nodes, while far-range pedestrians use deformable nodes based on human detection capability.This split distinguishes pedestrians according to whether the human pose-processing module detects them.
- Labeling Caveat: Dataset labels can group wheelchair users, skateboarders, and bicyclists as “cyclists,” while some annotations may be inaccurate.In Waymo, a person on a motorcycle may instead be labeled as a “vehicle,” potentially limiting generalization to scenarios with imperfect labels.
- Label Refinement: A preliminary GPT-4o experiment classified 60 individuals into pedestrians and personal-transportation-device users with 100% accuracy.The test used 30 individuals from each category, suggesting that accurate labels may be obtained relatively easily.