Source-linked AI summary
Panoptic Lifting for 3D Scene Understanding with Neural Fields
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, Peter Kontschieder
TL;DR
Panoptic Lifting addresses the challenge of producing coherent 3D panoptic understanding from noisy, view-inconsistent 2D masks. It learns a panoptic radiance field from posed images and machine-generated labels, with mechanisms for consistent instance lifting and label robustness. The method outperforms baselines across semantic and panoptic segmentation tasks and reports 43.6% versus 60.4% standard PQ on ScanNet for Mask2Former and Panoptic Lifting, respectively.
Problem
Single-image panoptic masks contain view-specific errors and do not preserve object identities across views, making coherent 3D lifting challenging.
Method
Panoptic Lifting learns a panoptic radiance field from posed images and machine-generated 2D panoptic masks, using semantic and instance fields plus robustness losses and confidence weighting.
Results
Panoptic Lifting outperforms baselines across datasets on semantic and panoptic segmentation without sacrificing view synthesis quality; on ScanNet, standard PQ is 60.4% versus 43.6% for Mask2Former.
Takeaways & Limitations
The method produces clean, coherent, 3D-consistent panoptic masks together with color and depth images for novel views from minimal input requirements.
Takeaways & Limitations
The method is limited to classes recognized by its pre-trained panoptic segmentation model and currently runs offline because preprocessing and neural-field optimization are lengthy.
Abstract
from arXiv · showhide
We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes. Once trained, our model can render color images together with 3D-consistent panoptic segmentation from novel viewpoints. Unlike existing approaches which use 3D input directly or indirectly, our method requires only machine-generated 2D panoptic segmentation masks inferred from a pre-trained network. Our core contribution is a panoptic lifting scheme based on a neural field representation that generates a unified and multi-view consistent, 3D panoptic representation of the scene. To account for inconsistencies of 2D instance identifiers across views, we solve a linear assignment with a cost based on the model's current predictions and the machine-generated segmentation masks, thus enabling us to lift 2D instances to 3D in a consistent way. We further propose and ablate contributions that make our method more robust to noisy, machine-generated labels, including test-time augmentations for confidence estimates, segment consistency loss, bounded segmentation fields, and gradient stopping. Experimental results validate our approach on the challenging Hypersim, Replica, and ScanNet datasets, improving by 8.4, 13.8, and 10.6% in scene-level PQ over state of the art.
1. Introduction
Panoptic Lifting addresses the difficulty of converting noisy, view-inconsistent 2D panoptic predictions into a coherent 3D scene representation. It learns a panoptic radiance field from posed images and machine-generated masks, using robustness mechanisms to produce consistent novel-view outputs.
- Motivation: 2D panoptic masks are insufficient for multi-view coherence because classifications and instance identities can vary across views.These inconsistencies make lifting machine-generated masks into a coherent 3D panoptic representation challenging.
- Approach: Panoptic Lifting represents a static scene as a panoptic radiance field trained from posed images and machine-generated 2D panoptic masks.The representation can render color, depth, semantics, and 3D-consistent instance information from novel viewpoints.
- Approach: A semantic MLP and a separate instance MLP add semantic classes and fixed class-agnostic surrogate identifiers to a TensoRF-based scene representation.The semantic field is supervised by 2D labels, while the instance field maintains 3D-consistent surrogate object identities.
- Robustness: Test-time augmentations provide confidence estimates, while segment consistency loss, bounded segmentation fields, and gradient stopping improve robustness to noisy labels.These components are designed to reduce the effects of inconsistent machine-generated segmentations.
- Outcome: The method directly lifts machine-generated 2D panoptic labels into a unified representation that supports clean, coherent, view-consistent segmentation from novel views.The approach is intended to support applications including novel panoptic view synthesis and scene editing.
2. Related Work
Prior work has used neural fields and other 3D representations to model semantic or panoptic scenes from images and spatial data. These approaches differ in supervision requirements, reliance on 3D detection and tracking, and support for photorealistic novel-view synthesis.
- Neural Radiance Fields: Neural radiance fields provide unified representations for scene appearance, geometry, and spatially varying properties such as semantics.They include coordinate-based networks and methods attaching parameters to explicit 3D structures.
- Semantic and Panoptic Neural Fields: Semantic NeRF methods fuse noisy 2D semantic segmentations into consistent volumetric models and enable novel-view synthesis of semantic masks.Later methods extend this direction with instance modeling and abstract visual features.
- Semantic and Panoptic Neural Fields: Panoptic Neural Fields uses RGB images alongside semantic segmentation, 3D bounding boxes, and object tracking from pre-trained networks.Its 3D detector dependence makes it sensitive to predicted-box errors, especially outside the training datasets of those networks.
- 3D Panoptic Segmentation: 3D panoptic segmentation has been studied on pre-computed meshes, voxel grids, and point clouds, as well as through simultaneous reconstruction and segmentation from images.The cited mesh, voxel-grid, and point-cloud methods do not provide photorealistic novel-view synthesis like NeRF-based methods.
3. Method
Panoptic Lifting builds a volumetric neural field that jointly models appearance, density, semantics, and instances from posed images with machine-generated 2D panoptic labels. Its rendering and training procedures lift inconsistent 2D predictions into 3D-consistent panoptic segmentations for novel views.
- Scene Representation: The representation models appearance and density with TensoRF, while lightweight MLP heads represent semantic classes and surrogate instance identifiers.The neural field assigns density, semantic distributions, surrogate-ID distributions, and color to 3D points.
- Rendering: Volumetric rendering produces color, semantic distributions, and surrogate-ID distributions along rays using the shared density field.The same rendering shorthand applies to all vector-valued fields provided by the panoptic radiance field.
- Novel-View Panoptic Rendering: For novel views, the most probable semantic class and surrogate identifier form a 3D instance identifier that remains consistent across the scene.This produces multi-view-consistent 2D panoptic segmentations when the scene is rendered from new viewpoints.
- Segmentation Supervision: Rendered semantic and instance fields are supervised using machine-generated 2D labels, with segmentation losses weighted by test-time augmentation confidences.Semantic supervision uses cross entropy, while mapped 2D instances supervise the rendered instance field through an NCE loss.
- Instance Lifting: Linear assignment maps view-specific 2D instance identifiers to compatible 3D surrogate identifiers using current instance predictions.The mapping is injective and solved as a linear assignment problem, allowing instance supervision across images.
- Training Objective: The total objective combines RGB, semantic, instance, and segment-consistency losses, while stopping segmentation gradients from reaching density prevents noisy labels from distorting geometry.The model is trained with separate ray batches for scene-wide RGB and semantic supervision and image-specific instance and consistency supervision.
4. Experiments
Experiments evaluate Panoptic Lifting on novel-view synthesis, semantic and panoptic segmentation, robustness to noisy labels, and scene editing across multiple datasets. The method outperforms baselines across datasets while preserving view-synthesis quality and supports object-level scene manipulation.
- Evaluation setup: The evaluation covers novel-view synthesis, novel-view panoptic segmentation, and scene editing on Hypersim, Replica, ScanNet, and in-the-wild captures.Ground-truth labels are reserved for evaluation, while training uses posed images and machine-generated labels.
- Results: Panoptic Lifting outperforms 2D and NeRF-based baselines across datasets on semantic and panoptic segmentation without sacrificing view-synthesis quality.The comparison uses shared images, poses, and Mask2Former-generated 2D labels; PNF also receives 3D boxes, including a ground-truth-box variant.
- Results: 43.6% and 60.4% are the standard PQ scores for Mask2Former and Panoptic Lifting on ScanNet, respectively.Panoptic Lifting also exceeds PNF by 10.6–15.7% in PQscene, while ground-truth boxes partially narrow PNF’s gap.
- Ablations: Disabling all robustness-oriented design choices reduces ScanNet performance by 8% mIoU and 11% PQscene.The ablations assess segment consistency loss, test-time augmentation, bounded segmentation fields, and blocked segmentation gradients.
- Ablations: Segment consistency loss counteracts label blending, while test-time augmentation produces smoother confidence estimates and improved masks.Disabling segment consistency loss causes the largest mIoU drop, and test-time augmentation improves both semantic metrics.
- Ablations: Bounded segmentation fields improve consistency, and blocking semantic and instance gradients to geometry prevents degradation in view-synthesis PSNR.Unbounded fields can predict inconsistent labels, while semantic supervision can otherwise push geometry toward incorrect solutions.
- Scene editing: After training, object instances can be deleted, duplicated, or manipulated with affine transformations in generated novel views.Deletion sets the density of points belonging to a selected instance to zero; duplication queries the original instance region from transformed rays.
- Limitations: The method is limited to classes recognized by its pre-trained panoptic model and currently runs offline because pose estimation, segmentation inference, and field optimization are lengthy.The paper identifies open-world segmentation and real-time SLAM integration as promising directions.
5. Conclusion
Panoptic Lifting lifts noisy machine-generated 2D panoptic labels into an implicit 3D volumetric representation that produces coherent, 3D-consistent masks with color and depth for novel views. Test-time augmentation further improves the panoptic inputs and provides per-pixel confidence scores.
- Panoptic Lifting lifts 2D machine-generated panoptic labels into an implicit 3D volumetric representation.
- The model produces clean, coherent, and 3D-consistent panoptic masks together with color and depth images for novel views.
- The method is reported as more robust to noise in machine-generated labels, improving results across datasets while supporting in-the-wild scenes.
- Test-time augmentation obtains improved panoptic segmentation masks and per-pixel confidence scores from Mask2Former.
A.2. Fusing Mask2Former predictions
The fusion procedure combines augmented Mask2Former segment predictions into a single panoptic segmentation with class labels, instance assignments, and confidence values. It clusters overlapping soft masks, averages masks and class probabilities within clusters, then assigns each pixel to the highest-confidence cluster.
- Candidate segments are represented by soft masks and class-probability distributions, then processed through segment clustering, cluster aggregation, and panoptic fusion.
- Segment clustering: Segments are connected when their soft-IoU exceeds a matching threshold, and connected components define the clusters.The threshold can be set to θ = 0.5.
- Cluster aggregation: Cluster masks and class probabilities are computed by averaging the masks and probabilities of their member segments.
- Panoptic fusion: For each cluster, the most likely class and corresponding probability are identified, and the cluster mask is scaled by that probability.
- Panoptic fusion: Each pixel is assigned to the cluster with the maximum scaled mask value, which becomes its confidence score.Pixels assigned to thing classes are partitioned into instances by cluster assignment.
B. Rendering Performance
Rendering performance is evaluated by measuring the time required to render 2048 rays on an NVIDIA RTX A6000 GPU. The authors attribute faster rendering than the baselines to Panoptic Lifting’s hybrid TensoRF representation.
- Rendering time is measured for a batch of 2048 rays on an NVIDIA RTX A6000 GPU.
- Panoptic Lifting renders faster than the baselines, which the authors attribute to its hybrid TensoRF representation.
C.1. Panoptic Lifting
Panoptic Lifting uses TensoRF for scene density and radiance, with separate view-independent MLPs for semantic classes and surrogate instance identifiers. Its experiments use publicly available Mask2Former predictions and compare against methods with differing additional input requirements.
- TensoRF models the scene density and radiance using a Vector-Matrix decomposition with specified density and appearance components and progressive grid resolution.The grid resolution increases from 128^3 to 192^3 during optimization.
- Two small view-independent MLPs model semantic class distributions and surrogate identifiers, with the identifier output limited to 50 classes in the experiments.
- The semantic and surrogate-identifier MLPs use MLPs rather than Vector-Matrix decompositions because of memory-size constraints.
- The experiments use publicly available Mask2Former code and models without retraining or fine-tuning, and all models use Mask2Former-generated labels.
- Panoptic Neural Fields is re-implemented without prior-based initialization because that initialization requires additional 3D datasets for the instanced classes.
- PNF experiments use a ScanNet-pretrained multiview detector for object boxes and require camera poses to be scaled and centered similarly to ScanNet training data.
D. Data
Experiments use posed image frames split into training and intermediate-view test sets, while evaluation labels remain excluded from training and model refinement. Predictions and ground-truth labels are mapped to ScanNet classes for dataset experiments.
- 75% of posed views are used for training and 25% for testing on intermediate sampled views.
- Ground-truth semantic and instance labels are used only for evaluation, not training or model refinement.
- Mask2Former predictions and ground-truth labels are mapped across datasets to ScanNet 21 classes.
- In-the-wild experiments use 31 ScanNet classes.
E. Additional Results
Additional results distinguish conventional per-frame PQ from scene-level consistency metrics and document the evaluated scenes and class mappings. In-the-wild scenes are evaluated qualitatively because ground-truth labels are unavailable.
- Conventional PQ comparisons with Mask2Former do not measure instance consistency across the scene because matching is performed independently per frame.
- SQscene and RQscene are additionally reported to evaluate scene-level semantic and recognition quality.
- The evaluation scenes are listed with their corresponding frame counts, while in-the-wild scenes support qualitative comparison only.
- Dataset experiments use ScanNet classes divided into stuff and thing types, with a separate class list for in-the-wild experiments.