PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Place Scene Labelling and Understanding with Implicit Scene Representation

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, Andrew J. Davison

arXiv:2103.15875v2cs.CV

TL;DR

Scene semantics cannot be learned fully self-supervised because classes are human-defined, while conventional learned methods may require data that does not match new scenes. The paper extends NeRF to jointly represent geometry, appearance, and semantics, using weak in-place supervision. Its scene-specific representation propagates sparse or noisy labels through smoothness and multi-view consistency, supporting dense semantic outputs and applications including denoising, super-resolution, and label fusion.

  • Problem

    Test-time distribution shifts and human-defined semantic classes make fully self-supervised semantic labelling of novel scenes impossible.

  • Method

    Semantic-NeRF extends NeRF with a joint implicit 3D representation of geometry, appearance, and semantics learned from sparse or noisy in-place labels.

  • Results

    The method generates complete, high-resolution semantic labels from partial, noisy, or low-resolution supervision and is evaluated on room-scale Replica and real-world ScanNet scenes.

  • Takeaways & Limitations

    NeRF’s smoothness and inherent multi-view consistency let sparse or noisy labels propagate for scene labelling, novel-view synthesis, denoising, super-resolution, and semantic fusion.

  • Takeaways & Limitations

    The paper focuses on cases where semantic supervision is sparse or noisy and leaves interactive selection of ambiguity-resolving labels for future work.

Abstract

from arXiv · show

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties. We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems.

1. Introduction

Scene-specific semantic understanding is needed because test-time scenes may differ from training data, yet human-defined semantics cannot be learned fully self-supervised. Semantic-NeRF addresses this by jointly learning scene geometry and semantics from sparse or noisy in-place labels.

  • 1. Introduction: Distribution shifts and open-set scenes limit conventional learned geometric and semantic prediction methods, motivating scene-specific self-supervision.Collecting annotated data or using semi-supervised learning can mitigate this issue but is not always feasible.
  • 1. Introduction: Semantic labels cannot be obtained purely self-supervised because semantic classes are human-defined and require some association with scene structures.Unsupervised clustering can group self-similar structures, but human labelling is still needed to assign semantic classes.
  • 1. Introduction: Semantic-NeRF jointly predicts geometry and semantics for a single scene using weak semantic supervision without geometric supervision.The network is trained on images from one scene with partial or noisy semantic labels.
  • 1. Introduction: Geometry self-supervision benefits semantic prediction through learned smoothness, coherence, and self-similarity, while multi-view consistency supports labels for substantially different views.These properties allow sparse semantic information to propagate across the scene.
  • 1. Introduction: The system is evaluated quantitatively and qualitatively on Replica and qualitatively on real-world ScanNet scenes, targeting dense labels from partial or noisy inputs.The stated applications include robotics scenarios where only limited in-situ labelling or imperfect single-view predictions are available.

2. Related Work

Prior 3D semantic systems use classical or learned representations, but many learning-based methods depend on extensive pre-training that may not cover unseen environments. Semantic-NeRF instead extends NeRF into a scene-specific implicit semantic representation learned from sparse or noisy annotations.

  • 2. Related Work: Classical 3D representations such as point clouds, surfels, voxels, and signed distance fields have limited ability to represent fine details in complex topologies.Volumetric representations also suffer from large memory requirements despite supporting parallel processing and convolutional networks.
  • 2. Related Work: Learning-based alternatives include compact latent-code representations that encode scene geometry or extend such representations to include semantics.Examples include GQN, CodeSLAM, and SceneCode.
  • 2. Related Work: Neural implicit representations provide continuous scene models that handle complicated topologies without discretisation error, with resolution determined by network capacity.Prior systems use MLPs or deep decoders to represent learned scenes or implicit signed distance functions.
  • 2. Related Work: Existing joint implicit appearance-and-semantics methods can synthesize novel-view semantic labels but typically use two-step semi-supervised training.Kohli et al. learn the representation on top of an SRN with a linear segmentation renderer.
  • 2. Related Work: Many prior learning-based representations require extensive pre-training, whereas scene-specific representations target minimum in-place labelling effort in unseen environments.Semantic-NeRF treats NeRF as a scene-specific 3D implicit representation and extends it with semantics learned from sparse or noisy annotations.

3. Method

Semantic-NeRF extends NeRF with scene-specific semantic prediction by rendering view-invariant labels alongside geometry and view-dependent colour. Joint photometric and semantic training uses sparse or noisy labels while preserving multi-view consistency.

  • 3.1. Preliminaries: NeRF represents scene density as a function of 3D position and colour as a function of position and viewing direction.Inputs are positional-encoded spatial coordinates and viewing directions.
  • 3.1. Preliminaries: NeRF estimates each pixel’s colour through hierarchical stratified sampling and numerical volume-rendering quadrature along camera rays.Samples lie between near and far bounds, with adjacent-sample distances determining the rendering weights.
  • 3.2. Semantic-NeRF: Semantic-NeRF appends a segmentation renderer and represents semantic logits as a view-invariant function of 3D position.Colour additionally receives viewing direction, whereas density and semantics depend only on spatial position.
  • 3.2. Semantic-NeRF: Rendered semantic logits are accumulated along rays using the same density-based sampling structure and converted to class probabilities with softmax.The renderer uses α and adjacent-sample distances to weight semantic contributions.
  • 3.3. Network Training: The network is trained from scratch with combined photometric and multi-class semantic losses using ground-truth, noisy, or partial labels.The semantic-loss weight is set to 0.04, and the losses encourage multi-view-consistent 2D renderings.
  • 3.3. Network Training: Each scene is trained individually with hierarchical coarse-to-fine volume sampling, positional encoding, Adam optimisation, and 200,000 iterations.Images are resized to 320x240, rays are batched in groups of 1024, and training uses a single RTX2080-Ti GPU.

4. Experiments and Applications

Experiments show that Semantic-NeRF learns coherent scene-wide semantic representations from sparse, noisy, coarse, or partial labels. Its multi-view-consistent joint representation supports view synthesis, denoising, super-resolution, annotation propagation, and semantic label fusion.

  • Semantic View Synthesis with Sparse Labels: Replica experiments show only marginal performance loss below 10% semantic-frame supervision, while two labelled key-frames still produce accurate labels from varied viewpoints.The main degradation comes from regions unobserved or occluded from the selected key-frames.
  • Semantic Fusion: Multi-view consistency is enforced during training, allowing inconsistent observations to be fused into a joint implicit 3D space for denoised labels.The training process itself functions as multi-view label fusion.
  • Semantic Fusion: When 90% of training pixels are randomly flipped, denoised labels retain accurate boundaries and fine detail, although entropy increases because noisy labels lack cross-view consistency.Void regions have the highest uncertainty because noisy void pixels are not optimized during training.
  • Semantic Fusion: Region-wise label corruption can also be corrected through multi-view consistency, with steady improvements that become harder as a larger fraction of labels is perturbed.The experiment changes whole chair-instance classes to simulate errors from obstructed or ambiguous views.
  • Semantic Fusion: Low-resolution semantic inputs produce accurate super-resolved labels for input and novel viewpoints, supporting coarse annotation and lightweight semantic mapping.The method is tested with sparsely sampled and interpolated low-resolution labels.
  • Semantic Fusion: In multi-view label fusion, the method achieves the highest improvement across all metrics while requiring posed images rather than the depth information required by two baseline methods.Baseline comparisons use ground-truth depth as a best-case data-association aid.

5. Conclusion and Future Work

The method generates complete, high-resolution scene semantics from partial, noisy, or low-resolution supervision, with practical relevance to scene understanding in new environments. Future work includes interactive labelling that requests labels resolving scene-wide ambiguity.

  • Semantic-NeRF generates complete and high-resolution semantic labels from partial, noisy, or low-resolution semantic supervision.
  • The method is intended for robotics and other applications requiring scene understanding when only limited labelling is possible.
  • Interactive labelling is proposed as future work, with the network requesting labels that most resolve semantic ambiguity for the whole scene.

Supplementary Material for In-Place Scene Labelling and Understanding with Implicit Scene Representation

Semantic-NeRF can reconstruct scene geometry and semantics in occluded or unobserved regions, while depth evaluation uses explicitly defined rendered and ground-truth quantities.

  • Semantic-NeRF predicts decent geometry and semantics in occluded regions and fills holes caused by unobserved regions to some extent.
  • The supplementary material presents semantic 3D reconstruction obtained using the learned scene-specific representation.
  • Depth metrics use n as the number of valid depth pixels, with d and dgt denoting rendered and ground-truth depths at testing poses.

A. Effects of Learning Semantics to Radiance and Geometry

Adding semantics produces no obvious difference in photometric and geometric reconstruction quality when projected to 2D on Replica scenes.

  • There is no obvious difference in photometric reconstruction quality between setups with and without semantics enabled.
  • There is no obvious difference in geometric reconstruction quality between setups with and without semantics enabled.
  • Rendered colour-image quality is measured with PSNR, while the depth-map metrics are defined in Table 5.

B. Semantic 3D Reconstruction from Posed Images

After training, an explicit 3D scene can be extracted from Semantic-NeRF’s implicit MLP representation, including geometry and attached semantic texture.

  • An explicit 3D scene can be extracted from the learned MLP to inspect the implicit 3D representation.
  • Geometric meshes are extracted by querying the MLP on dense 3D grids and applying marching cubes.
  • Attached semantic texture is rendered by using the negative normal direction of mesh vertices as ray-marching directions during volume rendering.

C. Network Architecture

The architecture uses positional encodings and fully connected layers to predict density, view-invariant semantics, and view-dependent radiance. Positional-encoding frequency affects label-propagation rendering quality, with L = 10 performing best empirically.

  • Network Architecture: 3D positional encodings feed both initial and intermediate fully connected layers before volume-density prediction.These layers use 256 neurons and ReLU activations.
  • Network Architecture: Additional 128-neuron fully connected layers predict view-invariant semantics and view-dependent radiance after viewing directions are merged.
  • Positional Encoding: L = 10 empirically performs best for label propagation, whereas L = 5 oversmooths renderings and L = 40 produces noisy interpolations.L controls the maximum positional-encoding frequency and affects rendering quality.

D. More Qualitative Results

The paper presents additional qualitative examples spanning semantic view synthesis, label denoising, and label super-resolution. It also identifies a quantitative evaluation of how semantic prediction affects appearance and geometry.

  • Semantic View Synthesis: Figure 10 provides additional qualitative results for semantic view synthesis.
  • Label Denoising: Figure 11 presents pixel-wise denoising of semantic labels with a 90% noise ratio.
  • Label Super-Resolution: Figure 12 presents label super-resolution results at 8× enlargement.
  • Quantitative Evaluation: Table 6 quantitatively evaluates the effects of predicting semantics on appearance and geometry using the Replica dataset.
Loading 2103.15875v2…