Source-linked AI summary
Neural Scene Graphs for Dynamic Scenes
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, Felix Heide
TL;DR
Existing neural rendering methods are largely limited to static scenes encoded in a single network and lack object-level dynamic decompositions. This paper learns a graph-structured neural representation with object transformations, radiance, and shared latent representations from video and tracking data. Experiments on simulated and real automotive data produce novel views and arrangements with unseen objects and poses, while also supporting inverse-rendering-based 3D object detection.
Problem
Existing neural rendering approaches encode static scenes in a single network and lack hierarchical representations and dynamic multi-object scene decomposition.
Method
The method learns a scene graph from video and tracking data, using decoupled object transformations, implicit radiance representations, and shared latent object descriptors.
Results
Experiments on simulated and real automotive data render photo-realistic novel views and novel dynamic arrangements with unseen object sets at unseen poses.
Takeaways & Limitations
The learned graph structure represents multiple dynamic and static scene elements and supports novel scene compositions and 3D object detection via inverse rendering.
Abstract
from arXiv · showhide
Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images. However, existing methods are restricted to learning efficient representations of static scenes that encode all scene objects into a single neural network, and lack the ability to represent dynamic scenes and decompositions into individual scene objects. In this work, we present the first neural rendering method that decomposes dynamic scenes into scene graphs. We propose a learned scene graph representation, which encodes object transformation and radiance, to efficiently render novel arrangements and views of the scene. To this end, we learn implicitly encoded scenes, combined with a jointly learned latent representation to describe objects with a single implicit function. We assess the proposed method on synthetic and real automotive data, validating that our approach learns dynamic scenes -- only by observing a video of this scene -- and allows for rendering novel photo-realistic views of novel scene compositions with unseen sets of objects at unseen poses.
1. Introduction
Existing neural rendering methods represent static scenes in a single network, whereas this work learns a scene graph for dynamic multi-object scenes from video and tracking data. The learned representation supports novel views, unseen object arrangements, and 3D object detection via inverse rendering.
- Existing neural rendering methods encode an entire static scene in one network and do not support hierarchical or dynamic scene representations.NeRF-W models transient elements but still does not provide the proposed dynamic scene decomposition.
- The method decomposes complex scenes into static and dynamic parts organized as a learned scene graph with decoupled object transformations and representations.Object representations are learned from video frames and tracking data, with instances of an object class sharing a volumetric representation.
- Experiments validate reconstruction and rendering of novel views and dynamic scene arrangements containing unseen object sets at unseen poses.The evaluation uses simulated and experimental data with labeled and generated tracking data.
- The proposed method also facilitates 3D object detection using inverse rendering.
2. Related Work
Related work combines differentiable neural rendering with implicit representations and hierarchical scene graphs. The proposed method addresses the interpretability gap by introducing a hierarchical scene graph with object-level implicit representations.
- Implicit Scene Representations and Neural Rendering: Unlike explicit graphics representations that model surfaces in scene-graph hierarchies, neural representations encode scene features as neural-network outputs.This implicit approach departs from traditional explicit scene modeling.
- Proposed Representation: The proposed method closes the interpretability gap by combining a hierarchical scene graph with object-level implicit representations.Figure 1 depicts nodes as boxes with local coordinate axes, transformation and scaling edges, latent descriptors, and unit-scaled bounding-box representation nodes.
- Implicit Scene Representations and Neural Rendering: Neural rendering learns implicit scene features with differentiable renderers from posed RGB images, enabling novel-view synthesis through ray casting or ray marching.Changing camera extrinsic parameters at test time supports novel views of a static scene.
- Scene Graph Representations: Traditional scene graphs organize objects as hierarchical leaf nodes, applying translation, rotation, and scaling transformations along edges in parent-local frames.The global transformation is derived from transformations through the hierarchy.
- Latent Class Encoding: Latent vectors appended to 3D query points let one network represent distributions of similar objects, with related work also learning latent codes through encoders or meta-networks.These descriptors support generalization across object classes.
3. Neural Scene Graphs
The neural scene graph models a scene hierarchically with camera, static, and dynamic nodes, while shared implicit representations and object-specific latent codes encode appearance and shape. Dynamic objects are transformed into local coordinate frames and rendered using pose- and location-conditioned radiance functions.
- Graph Definition: The scene graph is a directed acyclic hierarchy containing a camera, static node, dynamic nodes, representation models, latent object codes, and transformation or property-assignment edges.Object poses and locations are extracted by traversing transformations from the world root to the relevant representation node.
- Background Node: The static background maps position and viewing direction to volumetric density and emitted color, using sparse planes and Fourier features rather than a volumetric representation.The background function is implicitly stored in its network weights.
- Object Representation: Figure 2 distinguishes static and dynamic representation networks: both process ray points and directions, while dynamic models also receive an object latent descriptor and pose.The first stage outputs density σ and the second stage outputs color c.
- Latent Class Encoding: Dynamic objects are grouped into appearance classes that share representation weights, while a learned latent vector distinguishes individual object radiance fields.This reduces the number of separately trained models and supports shared object features across a class.
- Object Coordinate Frame: Object coordinates are transformed from global space into pose-aligned local frames and normalized by inverse bounding-box dimensions to learn size-independent class similarities.The local frame keeps the representation aligned with the object as its global pose changes.
- Dynamic Nodes: Dynamic object representations map latent codes, local points, and viewing directions to density and color, additionally conditioning appearance on global object position and pose.Pose is supplied only to the emitted-color pathway, while density remains pose-consistent.
4. Neural Scene Graph Rendering
The rendering pipeline represents dynamic scenes as hierarchical scene graphs, samples rays through background and object nodes, and predicts pixel colors by differentiable volumetric integration. Training jointly optimizes node representations from video frames, scene graphs, tracking transformations, and sampled-ray reconstruction loss.
- The scene graph organizes a dynamic scene and camera view hierarchically, with representation networks attached to leaf nodes.
- Ray Sampling and Rendering: Each camera ray is sampled at intersections with graph nodes, where representation functions predict volumetric density and directional color.Static and dynamic nodes provide the corresponding radiance-field outputs.
- Ray Sampling and Rendering: The static node uses multiple planes parallel to the initial camera image plane, replacing raymarching with plane intersections for greater efficiency.The planes are evenly spaced between near and far clipping distances.
- Ray Sampling and Rendering: Dynamic nodes are sampled by translating each ray into an object’s local frame, testing axis-aligned bounding-box intersections, and evaluating equidistant points inside each intersection.A small number of dynamic-object samples is intended to preserve accuracy while maintaining short rendering times.
- Ray Sampling and Rendering: The renderer orders all sampled points and model outputs along each ray, then estimates pixel color with numerical quadrature of the volume-rendering integral.Sampling includes dynamic-node intersections and static-node planes.
- Joint Scene Graph Learning: Training uses image–scene-graph tuples, tracking-derived transformations, randomly sampled rays, squared pixel-color error, and Gaussian-prior regularization of object latent descriptors.Gradients are backpropagated through trainable nodes intersected by each ray batch, with sampling balanced across nodes.
- Joint Scene Graph Learning: KITTI renderings show decomposed background and object representations, while randomly sampled nodes and translations produce novel scene compositions.The figure illustrates medium and higher sampling densities for novel renderings.
5. Experiments
The experiments evaluate neural scene graphs on automotive video data by manipulating learned graphs, synthesizing novel views and arrangements, and comparing reconstruction quality with implicit-rendering baselines.
- Experimental Setup: The method is trained on KITTI sequences containing up to 90 time steps, two camera perspectives, and up to 12 dynamic objects.The experiments also assess whether tracking outputs can replace dataset annotations.
- Foreground-Background Decomposition: Without additional supervision, the learned graph separates static background components from dynamic objects while retaining object shadows in dynamic representations.Removing dynamic nodes renders the background, while removing the static node renders the dynamic parts.
- Novel Scene Graph Compositions and View Synthesis: The method synthesizes unseen views after approximately 2 m of ego-camera motion while handling occlusions from traffic lights and signs.Other scene-graph nodes remain fixed during this camera translation.
- Scene Graph Manipulations: Changing node transformations preserves global illumination cues, moving reflections and specular highlights consistently when objects rotate or translate.The representation encodes illumination through image color as a function of object location and viewing direction.
- Novel Scene Graph Compositions and View Synthesis: Novel scene graphs combine sampled objects, graph edges, and new transformations on observed road trajectories while preventing object collisions.These arrangements and translations were not observed during training.
- Quantitative Validation: The proposed method outperforms SRN, NeRF, and NeRF + time on all reported reconstruction metrics and improves temporal consistency for moving objects.NeRF + time improves quality but still lacks detail and produces blurry, uncertain predictions; the comparison includes PSNR, SSIM, LPIPS, tOF, and tLP.
6. 3D Object Detection as Inverse Rendering
The method formulates 3D object detection as inverse rendering over learned scene graphs. It optimizes object poses and dimensions to synthesize an image matching the observation.
- 3D object detection is formulated as image synthesis over the space of learned scene graphs.
- The selected scene graph is the one whose synthesized image has minimum distance to the observed image.
- Anchor positions, object box positions, and latent object codes are optimized using ℓ1 image loss against the observed image.
- The resulting detections recover object poses and bounding-box dimensions from the optimized scene graph.
7. Discussion and Future Work
The paper represents dynamic multi-object scenes with graph-structured neural representations and validates novel object arrangements on simulated and real data. It also identifies training-data variation and amount as a boundary on representation quality.
- The method learns continuous graph-structured representations of multiple dynamic and static scene elements from video and annotated tracking data.
- Experiments on simulated and real data generate photo-realistic novel object arrangements using the learned graph structure.
- The learned representation quality is bounded by the variation and amount of training data.