Source-linked AI summary
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, Thomas Funkhouser
TL;DR
Existing neural scene representations lack a unified object-aware representation for semantic understanding of dynamic scenes from images alone. PNF decomposes scenes into object and background fields, jointly optimizes their parameters and poses, and supports several reconstruction, segmentation, editing, depth, and view-synthesis tasks. Experiments on real-world dynamic scenes demonstrate its utility across these tasks.
Problem
Existing neural scene representations lack a unified object-aware approach that supports semantic 3D understanding of dynamic scenes from color images alone.
Method
PNF uses separate instance-specific object MLPs and a semantic background MLP, initialized with category priors and jointly optimized with oriented bounding boxes from image-based supervision.
Results
PNF achieves state-of-the-art quality across multiple tasks and benchmarks on KITTI and KITTI-360, including novel-view synthesis and panoptic segmentation.
Takeaways & Limitations
The panoptic-radiance field can be queried or rendered for color, density, category, instance, segmentation, depth, scene editing, and novel views.
Takeaways & Limitations
PNF is compute-intensive, limited to rigid moving objects, and does not handle missing or duplicate detections or incorrect class predictions.
Abstract
from arXiv · showhide
We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.
1. Introduction
PNF targets full 3D understanding of dynamic scenes by decomposing them into object instances and background while jointly learning geometry, appearance, semantics, and poses from color images. The unified representation supports multiple scene-understanding and synthesis tasks.
- Motivation: PNF addresses the gap between image-level recognition and full 3D scene understanding for dynamic things and background stuff.The target includes geometry reconstruction, radiance, decomposition, and category and instance labels for 3D points.
- Representation: PNF represents each object instance with a separate MLP inside a mobile oriented 3D bounding box and represents background stuff with a semantic MLP.Together, these fields describe density, color, category, and instance labels over time.
- Representation: Category-specific meta-learned initialization provides object shape and appearance priors while allowing smaller, faster instance-specific MLPs.Separating category priors from instance-specific details is intended to speed inference in scenes with many objects.
- Optimization: The method uses predicted camera parameters, 2D segmentations, and 3D object detections, then jointly optimizes bounding boxes and MLP parameters through analysis-by-synthesis losses.Rendered outputs are compared with the available image and segmentation information.
- Evaluation: PNF is evaluated on KITTI and KITTI-360 for 3D panoptic reconstruction, scene editing, novel-view synthesis, segmentation, depth, and related outputs.The representation can render semantic, panoptic, depth, and color images from observed or novel views.
2. Related Work
Prior work covers image-based reconstruction, multi-view mapping, semantic neural fields, dynamic NeRFs, and object-decomposed radiance fields. PNF is positioned as addressing missing combinations of panoptic output, dynamic objects, object specificity, and pose optimization.
- Research Landscape: Related work spans single-image 3D reconstruction, multi-view reconstruction and segmentation, and neural fields.The paper organizes prior methods into three broad categories.
- Reconstruction and Segmentation: Single-image and multi-view systems provide reconstruction or segmentation capabilities but commonly require supervision, static scenes, or discrete voxel representations.PanopticFusion and ATLAS, for example, are described as limited by static-scene assumptions, supervision requirements, or voxel-grid resolution.
- Comparison: Table 1 compares prior methods by semantic segmentation, object decomposition, panoptic segmentation, dynamic-object handling, bounding-box optimization, and novel-view synthesis.These properties define the capability axes used for the related-work comparison.
- Neural Fields: NeRF maps positions and directions to density and radiance for novel-view synthesis but is limited to static scenes and scene-specific training from scratch.SemanticNeRF adds semantic labels, while related semantic neural fields remain neither object-aware nor dynamic.
- Dynamic Neural Fields: Dynamic NeRFs model time-varying deformation for selected deforming subjects, whereas PNF considers scenes containing many moving objects.The distinction is the scene-level focus on multiple moving objects.
- Object-Aware Neural Fields: Object-decomposed NeRF systems separate foreground objects from background but do not provide the full panoptic, dynamic, and instance-specific capabilities targeted by PNF.Earlier approaches use shared or conditional representations and latent codes for object instances or categories.
3. Method
PNF represents scenes as a panoptic-radiance field composed of separate dynamic object functions and a semantic background function, trained and rendered through NeRF-style volumetric processing.
- 3.1. Scene Representation: PNF represents each object with a separate MLP inside a dynamic oriented bounding box, alongside a larger MLP for background stuff.Queries use position, view direction, and time, producing color, density, semantic labels, and instance labels.
- 3.1. Scene Representation: The final field aggregates contributions from object bounding boxes and defaults to the stuff output when no object box intersects the queried point.Semantic outputs use object-class one-hot logits, while instance outputs identify detected objects and assign zero to stuff.
- 3.2. Rendering Panoptic-Radiance Fields: PNF renders RGB, depth, semantic, and instance outputs by sampling 1024 points with jitter along each ray and alpha-compositing the queried channel values.The rendering weights are determined by over-compositing sample opacities along the ray.
- 3.3. Model Losses and training: Training jointly optimizes network parameters and object tracks using RGB reconstruction loss and semantic cross-entropy against predicted 2D segmentations.Object tracks are optimized because initial bounding boxes, including ground-truth boxes, may be noisy.
4. Evaluations
PNF is evaluated across view synthesis, reconstruction, segmentation, depth estimation, object decomposition, and scene editing on KITTI-family datasets. It achieves strong performance while preserving dynamic objects and enabling object-level manipulation.
- Evaluation scope: PNF is evaluated on view synthesis, reconstruction, 2D panoptic segmentation, 2D depth prediction, and scene editing across KITTI, Virtual KITTI, and KITTI-360.The datasets contain challenging outdoor dynamic scenes.
- Novel-view synthesis: PNF is the best-performing method for both rendered semantic segmentation and color images on the KITTI-360 leaderboard.Table 2 reports results for novel-view color and semantic synthesis.
- Panoptic segmentation: 74.28 mIoU is achieved for novel-view semantic segmentation on KITTI-360, outperforming two-stage synthesis-and-segmentation baselines.The model renders segmentation directly from the unified representation and remains temporally consistent in dynamic scenes.
- Panoptic segmentation: +9.2 mIoU over SemanticNeRF is reported on dynamic KITTI segmentation, where non-object-aware models cannot model dynamic objects.PNF also improves upon single-image segmentation by fusing information from multiple views.
- Depth and reconstruction: PNF reconstructs moving-car shapes sharply, whereas standard NeRF models would miss or blur those cars.The result is attributed to object-aware handling of dynamic scenes and instance-specific object MLPs.
- Object decomposition: PNF disentangles objects from background stuff more effectively than NSG, avoiding traffic-sign entanglement and reconstructing bus windows as translucent.The decomposition is visualized by rendering objects without the background.
- Scene editing: PNF supports scene editing by removing, adding, cloning, and transforming objects through their separate 3D radiance fields and poses.Virtual KITTI examples clone all cars to be the same or independently rotate vehicle objects.
5. Limitations
PNF remains constrained by computational cost, simplified light transport, detection quality, and its restriction to rigid moving objects.
- Limitations: PNF is compute-intensive and currently suited only to offline applications.The authors expect advances in neural rendering to alleviate speed issues.
- Limitations: PNF does not model complex light transport such as shadowing under object motion.
- Limitations: PNF can correct noisy bounding-box poses but does not handle missing or duplicate detections or incorrect class predictions.
- Limitations: PNF is restricted to scenes with rigid moving objects and does not handle deformable objects.
6. Conclusion
PNF represents dynamic scenes as 4D panoptic radiance fields that expose color, density, instance, and category information over time. This representation supports rendering-based scene editing, view synthesis, and panoptic segmentation, with state-of-the-art results reported on challenging KITTI scenes.
- Conclusion: PNF decomposes scenes into object-instance and background MLPs to learn a 4D panoptic radiance representation from images alone.
- Conclusion: The representation can be queried for color, density, instance, and category labels at any 3D point over time.
- Conclusion: Scene editing, view synthesis, and panoptic segmentation are derived by rendering the representation from desired views.
- Conclusion: Experiments on KITTI scenes demonstrate state-of-the-art novel-view synthesis and panoptic segmentation for challenging outdoor scenes with multiple dynamic objects.
A. Additional Model Details
This section introduces additional training and inference details for the proposed panoptic neural field model.
- Additional Model Details: The section provides additional details about training and inference for the proposed panoptic neural field model.
A.1. Network Architecture and Training Details
PNF uses separate network designs for stuff and thing fields, with view-dependent color and position-dependent density and semantics. Training uses stratified ray sampling, scaled semantic loss, rotation constraints, and category-specific initialization produced through federated averaging.
- Network architecture: The stuff MLP has 8 hidden layers of width 256, while each thing MLP has 4 hidden layers of width 128.Position encoding uses 10 frequencies for stuff and 6 for thing MLPs.
- Network architecture: Density and semantic outputs depend only on position, while color additionally depends on view direction.This encodes position-only structure and semantics for the background and individual objects.
- Training details: PNF replaces hierarchical sampling with stratified sampling and uses one MLP set, including 1024 samples per ray on KITTI.This reduces memory relative to coarse-and-fine NeRF sampling.
- Training details: The objective combines color and semantic losses, scales semantic loss downward, and constrains optimized rotations to valid SO(3) matrices.The semantic loss uses softmax cross-entropy, whereas RGB uses mean squared error.
- Initialization: Thing MLP weights use category-specific initialization for cars and vans, meta-learned from rendered ShapeNet car images.A simplified federated averaging procedure realizes the learned initialization.
- Initialization: FedAvg trains each object-instance client on ray bundles for local epochs, updates MLP weights with SGD, and returns them to the server.The algorithm uses minibatch size B, local epochs E, and learning rate η before server aggregation.
B. Additional Results
Additional experiments evaluate PNF on KITTI and KITTI-360 for novel-view color, semantic, and depth synthesis, scene editing, and learned initialization on ShapeNet.
- Evaluation scope: The additional results examine novel-view synthesis of color, semantics, and depth images alongside scene editing.These results extend evaluation beyond the core experiments.
B.1. Novel view renderings
Additional renderings show PNF-generated semantic, depth, and color views from KITTI-family scenes, together with object-level scene edits and initialization comparisons on ShapeNet.
- Novel-view renderings: KITTI and KITTI-360 results render semantic segmentation and depth from forward-facing camera data captured at approximately 10Hz and 5Hz, respectively.The KITTI-360 and KITTI figures report accurate reconstruction and segmentation, including thin structures and scenes with moving cars.
- Scene editing: The scene-editing figure compares original and edited novel-view color renderings, including object insertion, cloning, and other object manipulations.The edits include adding or removing objects, changing 3D poses, and replicating thing MLP parameters across objects.
- Initialization comparisons: ShapeNet comparisons evaluate standard Xavier initialization against learned federated-averaging initialization after two full training epochs.The dense-view comparison uses at least 50 input views, while the sparse-view comparison uses one or three views.
- Novel-view renderings: PNF renders semantic overlays, depth, and color images for novel viewpoints using the learned panoptic-radiance-field representation.The reported novel-view results use only forward-facing images for the test sequences.
B.3. Benefits of our learned initialization
Category-specific priors are meta-learned from ShapeNet and used to initialize instance-specific object fields. The learned initialization improves efficiency, convergence, and reconstruction completeness, especially with sparse observations.
- Motivation: The category-specific priors address the limited viewpoints commonly available for objects such as cars in self-driving scenes.Sparse observations make prior knowledge important for completeness and accurate reconstruction.
- Prior learning and efficiency: Meta-learned ShapeNet priors are distilled into initialization for novel scenes, allowing inference networks to focus on the scene’s individual object instances.The resulting thing MLP uses 10x fewer FLOPs per object than NSG while achieving better dynamic-scene image reconstruction in Tab. 3.
- Sparse-view reconstruction: The learned initialization provides faster convergence and better completeness when objects are reconstructed from sparse partial observations.These benefits are demonstrated on ShapeNet using rendered car images.
- Dense-view reconstruction: With at least 50 input views, learned initialization offers clear convergence benefits over standard Xavier initialization after two full training epochs.Figure 14 provides the dense-view qualitative comparison.
- Sparse-view reconstruction: The learned initialization advantage is more pronounced with few sparse input views than with dense observations.Figure 15 compares reconstructions from one or three input views against standard initialization.
- Sparse-view reconstruction: Using one image, the learned initialization reconstructs novel object instances with more complete geometry from partial views.The category-specific priors supply information beyond the directly observed object surface.
C. Potential Negative Societal Impact
PNF is an intermediate representation for comprehensive 3D scene understanding with applications that may have beneficial or negative societal impacts. The authors identify surveillance and synthetic-media misuse risks and describe mitigation measures, while noting that all possible applications cannot be anticipated.
- PNF’s comprehensive 3D scene understanding representation could support applications with either beneficial or negative societal impact.
- Because PNF tracks objects and people, it could support crowd monitoring and traffic-density reporting but also surveillance systems.
- The authors plan to limit code use to academic research and label generated images as synthetic, while acknowledging mitigation is difficult.
- PNF’s view synthesis and scene editing could enable fabricated or altered imagery used for fake news or propaganda.