PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, Yiyi Liao

arXiv:2203.15224v2cs.CV

TL;DR

Pixel-wise urban-scene annotation is costly, motivating methods that transfer coarse 3D labels into dense 2D semantic and instance labels. Panoptic NeRF fuses coarse 3D primitives with noisy 2D predictions through NeRF-based 3D inference and rendering, outperforming existing label-transfer methods while producing multi-view-consistent labels.

  • Problem

    Pixel-wise segmentation annotation is labor-intensive and costly, creating a need for efficient transfer from coarse 3D annotations to dense 2D labels.

  • Method

    Panoptic NeRF uses dual semantic fields to combine coarse 3D bounding primitives with noisy 2D predictions, optimize geometry, and render per-pixel labels from 3D.

  • Results

    Panoptic NeRF outperforms existing label-transfer methods in semantic and instance prediction accuracy and provides multi-view-consistent labels by design.

  • Takeaways & Limitations

    The method supports efficient development of large-scale, densely labeled autonomous-driving datasets and continuous 3D-consistent panoptic scene representations.

  • Takeaways & Limitations

    Training takes 4 hours per scene, and the method is limited to static scenes rather than dynamic scenes.

Abstract

from arXiv · show

Large-scale training data with high-quality annotations is critical for training semantic and instance segmentation models. Unfortunately, pixel-wise annotation is labor-intensive and costly, raising the demand for more efficient labeling strategies. In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives. Our method utilizes NeRF as a differentiable tool to unify coarse 3D annotations and 2D semantic cues transferred from existing datasets. We demonstrate that this combination allows for improved geometry guided by semantic information, enabling rendering of accurate semantic maps across multiple views. Furthermore, this fusion process resolves label ambiguity of the coarse 3D annotations and filters noise in the 2D predictions. By inferring in 3D space and rendering to 2D labels, our 2D semantic and instance labels are multi-view consistent by design. Experimental results show that Panoptic NeRF outperforms existing label transfer methods in terms of accuracy and multi-view consistency on challenging urban scenes of the KITTI-360 dataset.

1. Introduction

Panoptic NeRF addresses the cost and difficulty of pixel-wise annotation by transferring coarse 3D annotations and noisy 2D predictions into accurate, multi-view-consistent 2D panoptic labels. Its joint geometry and semantic optimization improves sparse-view geometry, resolves label ambiguity, and filters prediction noise.

  • Motivation: Pixel-wise annotation can take up to 1.5 hours per street-scene image, whereas coarse 3D labeling reduces annotation time to 0.75 minutes per image.Coarse 3D annotation is also easier for separating instances in cases such as pedestrians in front of buildings.
  • Related approaches: Existing methods either manually post-process coarse 3D annotations or combine 3D and noisy 2D cues with CRFs and intermediate 3D reconstructions.The CRF-based approach projects non-occluded 3D points into 2D before performing inference in image space.
  • Challenges: Vanilla NeRF produces inaccurate label maps when sparse-view geometry is wrong, while overlapping 3D primitives create additional label ambiguity.Panoptic NeRF targets both challenges by combining coarse 3D annotations with noisy 2D semantic predictions.
  • Method: Panoptic NeRF uses semantically guided geometry optimization followed by joint geometry and semantic optimization to fuse 3D primitives with noisy 2D predictions.The fixed semantic field first guides geometry, and the learned semantic field then refines semantic rendering.
  • Results: The fusion resolves primitive-label ambiguity, removes noise from 2D predictions, and renders globally consistent 2D instance maps across multiple frames.Each object receives a unique instance index determined by the 3D bounding primitives.
  • Results: Panoptic NeRF outperforms existing 3D-to-2D and 2D-to-2D label-transfer methods on KITTI-360 urban scenes.The method provides a route toward efficiently developing large-scale, densely labeled autonomous-driving datasets.

2. Related Work

Related work addresses efficient urban-scene labeling through 2D-to-2D propagation, 3D-to-2D transfer, and coordinate-based neural representations. Existing 3D-assisted approaches reduce labeling effort but face challenges in producing dense 2D labels.

  • Urban Scene Segmentation: Urban-scene segmentation methods rely on large-scale training data, but pixel-level and especially instance-level image annotation is time-consuming and labor-intensive.KITTI-360 shows that annotating scenes in 3D can significantly reduce annotation time, although transferring coarse 3D labels to 2D remains challenging.
  • Label Transfer: 2D-to-2D label-transfer methods reduce workload by propagating labels across 2D images, whereas 3D-to-2D methods exploit additional 3D information.The paper focuses on efficient labeling of video sequences.
  • Label Transfer: Non-local multi-field CRF methods jointly infer over 3D point clouds and 2D pixels but require reconstructing a 3D mesh for dense image-space labels.These methods represent a 3D-assisted alternative within the 3D-to-2D labeling category.
  • Coordinate-based Neural Representations: Coordinate-based neural representations have been applied to 3D reconstruction, novel-view synthesis, and 3D generative modeling, while NeSF studies generalizable semantic-field learning.This work uses coordinate-based representations to estimate scene semantics.

3. Background

The paper formulates Panoptic NeRF as a 3D scene representation that uses sparse posed RGB images, coarse labeled 3D primitives, and noisy 2D semantic predictions to produce dense multi-view labels. NeRF supplies the volume-rendering mechanism that maps continuous scene fields to image pixels.

  • NeRF: NeRF models a 3D scene as a continuous neural radiance field mapping a 3D coordinate and viewing direction to density and RGB color.A camera ray is represented as r(t) = o + td.
  • NeRF: Volume rendering computes each pixel’s color from densities and colors sampled along a camera ray using transmittance and sample intervals.The rendering process enables learning the field from posed 2D RGB images.
  • Problem Formulation: Panoptic NeRF transfers coarse 3D bounding primitives into dense 2D semantic and instance labels using sparse posed RGB images as additional input.The primitives cover the scene as cuboids, ellipsoids, and extruded polygons.
  • Problem Formulation: Each 3D primitive carries a stuff-or-thing semantic label, and thing primitives additionally have unique instance IDs.A pretrained semantic segmentation model supplies a 2D semantic prediction for each RGB image.

4. Methodology

Panoptic NeRF uses fixed and learned semantic fields within NeRF to improve geometry, fuse coarse 3D primitives with noisy 2D cues, and render consistent semantic and panoptic labels.

  • Dual Semantic Fields: Panoptic NeRF maps each 3D point to density, color, and two semantic distributions, then volume-renders both distributions along camera rays.The fixed field is determined by 3D bounding primitives, while the learned field is produced by a semantic head.
  • Dual Semantic Fields: The fixed semantic field assigns one-hot labels inside uniquely enclosing primitives and equal probabilities at overlaps, preserving geometry guidance but not resolving ambiguity.The learned semantic field uses a softmax semantic head to represent categorical distributions.
  • Semantically-Guided Geometry Optimization: Applying semantic loss to the fixed field updates density directly, allowing noisy 2D predictions to improve geometry and object-boundary estimation under sparse views.Weak stereo-based depth supervision mitigates wrong pseudo-labels, while semantic guidance improves boundaries beyond depth supervision alone.
  • Joint Geometry and Semantic Optimization: Joint optimization of geometry and the learned semantic field addresses overlapping-region ambiguity, while per-point supervision targets unique labels and sufficiently dense object surfaces.The per-point mask is active when a point has a unique 3D label and density above σth.
  • Training and Sampling: The total objective combines semantic, photometric, and depth losses, and the method samples within primitives to focus computation on non-empty regions.Training uses one model per scene for 80,000 iterations with Adam and a density threshold σth = 0.1.
  • Panoptic Segmentation: Panoptic labels use the learned semantic distribution for stuff classes and a deterministic instance distribution from bounding primitives for thing classes.The instance field is one-hot when a point is uniquely enclosed by a thing primitive; thing primitives rarely overlap, supporting reliable instance rendering.

5. Experiments

Experiments on KITTI-360 evaluate semantic, panoptic, novel-view, and ablation performance against label-transfer baselines. Panoptic NeRF achieves stronger semantic and panoptic transfer, supports novel-view synthesis, and relies on semantic guidance and dual-field components, while per-scene optimization limits scalability.

  • Dataset: Experiments use KITTI-360, with manually annotated frames from five static suburbs divided into 10 scenes for evaluation.The training data comprise 128 consecutive posed stereo-image pairs per scene.
  • Label Transfer: Semantic Label Transfer: Panoptic NeRF achieves the highest mIoU and Acc among the evaluated baselines.It improves mIoU over 3D-2D CRF by 1.6 percentage points, from 79.5% to 81.1%.
  • Label Transfer: Panoptic Label Transfer: Panoptic NeRF outperforms 3D-2D CRF for both things and stuff classes.Qualitative results show stronger handling of overexposed building regions, where intermediate mesh reconstruction can be poor.
  • Novel View Label Synthesis: Panoptic NeRF renders RGB images and panoptic labels at novel viewpoints, unlike 3D-2D CRF.Its labels are designed to remain multi-view and spatio-temporally consistent.
  • Ablation Study: Ablations show that removing 2D semantic supervision, fixed semantic fields, depth-related components, or the proposed sampling strategy degrades geometry and semantic estimation.The full model produces sharper edges, whereas removing fixed semantic fields yields over-smoothed object boundaries.
  • Ablation Study: Using different 2D pseudo-ground-truth models, Panoptic NeRF consistently outperforms the corresponding pseudo-ground-truth labels.Pre-trained Cityscapes models without fine-tuning still produce promising results in the reported mIoUsub comparison.
  • Limitations: Panoptic NeRF performs per-scene optimization, requiring four hours to train on one scene, and is limited to static scenes.The authors identify training-time reduction and extension to dynamic scenes as future directions.

6. Conclusion

Panoptic NeRF performs 3D-to-2D label transfer by inferring in 3D and rendering per-pixel semantic and instance labels. It combines coarse 3D primitives with noisy 2D predictions, improves geometry and label quality, and supports novel-view synthesis.

  • Panoptic NeRF infers in 3D space and renders per-pixel semantic and instance labels for 3D-to-2D transfer.
  • Combining coarse 3D bounding primitives with noisy 2D predictions through dual semantic fields improves geometry from sparse views and resolves label noise.
  • The method enables semantic and instance label synthesis at novel viewpoints while providing a 3D-consistent continuous panoptic scene representation.

A.1. Network Architecture

Panoptic NeRF uses a NeRF-style network with positional encoding for 3D locations and viewing directions, while its semantic field predicts view-independent distributions. Sampling is restricted to intersections with nearby convex bounding primitives.

  • The network takes normalized 3D locations and viewing directions, mapping both through NeRF-style positional encoding.The 3D location elements are normalized to [−1, 1].
  • The semantic field depends only on 3D location, producing view-independent semantic distributions through semantic logits and a softmax layer.The positional-encoding levels are L = 15 for γ(x) and L = 4 for γ(d).
  • The model samples points inside intersected convex bounding primitives, sorting them from near to far and retaining the first 10 for efficiency.Each ray intersects a convex primitive twice, defining its sampling interval; the remaining primitives are likely occluded.

A.3. Evaluation Metric

The evaluation uses standard semantic metrics alongside multi-view consistency and panoptic quality measures. Consistency is computed from LiDAR-matched pixels, while small ground-truth regions are excluded from panoptic matching.

  • Semantic performance is evaluated using mean intersection-over-union and pixel accuracy following standard practice.
  • Multi-view Consistency: Multi-view consistency matches pixels across consecutive frames using LiDAR depth maps and counts pairs whose 3D distance is below 0.1 meters.The metric is the number of consistent pairs divided by all matched pairs.
  • Multi-view Consistency: Cuboids, ellipsoids, and locally extruded 3D planes are treated as convex primitives for the evaluation setup.
  • Panoptic Quality: Panoptic quality uses void labels for ground-truth regions smaller than 100 pixels, preventing segment matching in those regions.This handles imprecise distant labels and small noisy thing regions.

A.4. Training and Inference

Training combines photometric, depth, and semantic losses, with semantic and depth supervision applied mainly to left images for the stated comparison setup. Stereo-derived depth is filtered for consistency and range, while inference can use other views.

  • The total loss contains three semantic losses, a photometric loss, and a depth loss during per-scene optimization.
  • Photometric supervision uses posed stereo images, whereas 2D semantic losses are applied to left images to ensure fair comparison with 3D-2D CRF.
  • The 3D semantic and depth losses are defined on points and images from the left views, with right-view depth providing marginal information gain.
  • Evaluation compares left views with manual ground truth, although the method is not restricted to left-view inference and supports right and novel views.
  • Weak depth supervision comes from stereo Semi-Global Matching, filtered by left-right and five-frame consistency checks and limited to predictions within 15 meters.

B.2. LiDAR Depth for Evaluation

The evaluation examines depth, semantic transfer, stereo consistency, and novel-view label synthesis. Qualitative results support improved boundaries, baseline performance, cross-view consistency, and unseen-view synthesis.

  • Depth evaluation: The evaluation uses accumulated LiDAR observations projected into each frame to compare rendered depth maps against LiDAR measurements without using LiDAR as input.This preserves evaluation against LiDAR while allowing operation in settings without LiDAR observations.
  • Evaluation setup: Ambiguous KITTI-360 ground-truth regions were manually relabeled to improve label quality for fair comparison.
  • Ablation: Adding the 2D semantic loss to the fixed semantic field further improves object boundaries beyond depth loss alone.Depth supervision is weak at far regions, while semantic errors incur a stronger penalty for small depth deviations.
  • Semantic and panoptic transfer: Our method qualitatively outperforms all baselines, consistently with the quantitative results in Table 1.Additional comparisons include semantic and panoptic label transfer on unlabeled 2D frames.
  • Stereo label transfer: Stereo label transfer remains consistent on both left and right views despite supervision using pseudo ground truth only on left views.
  • Novel-view synthesis: Novel-view experiments using photometric supervision only on left images produce promising appearance and label synthesis results on right-view images.The evaluation tests synthesis on views not directly used for photometric supervision.

C.5. Analysis of 3D-2D CRF

The analysis compares Panoptic NeRF with 3D-2D CRF across semantic, panoptic, stereo, and novel-view transfer. It also identifies incomplete reconstruction and overlapping primitives as important boundary-related limitations.

  • 3D-2D CRF limitations: 3D-2D CRF can lack 3D information where LiDAR produces incomplete mesh reconstruction, forcing greater reliance on 2D image cues.Building sides are difficult to scan, leaving incomplete mesh regions.
  • 3D-2D CRF limitations: 3D-2D CRF fails in the illustrated case at overexposed image regions when 3D information is missing.
  • Failure cases: Panoptic NeRF struggles with accurate instance boundaries where two 3D bounding primitives overlap, including spatially connected buildings.The deterministic instance field defined by the primitives constrains boundary recovery in these regions.
  • Qualitative comparisons: The figures compare semantic transfer against 3D-2D CRF on labeled and unlabeled frames, with panoptic colors not directly matching without 2D ground truth.
  • Additional evaluations: Stereo results are shown as blended semantic outputs for left and right views, while novel-view synthesis compares ground-truth RGB, rendered RGB, and rendered semantic maps.
  • 3D-2D CRF visualization: The 3D-2D CRF figure presents input RGB images, reconstructed mesh depth, and resulting panoptic label transfer outputs.
  • Failure cases: A failure example shows correct semantic labeling but an incorrectly segmented boundary between adjacent buildings in the panoptic map.
Loading 2203.15224v2…