PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views

Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, Federico Tombari

arXiv:2404.03650v1cs.CV

TL;DR

OpenNeRF addresses limitations in open-set 3D segmentation caused by closed-set training and imperfect alignment between image-based VLM features and explicit 3D representations. It distills pixel-aligned CLIP features into a NeRF and uses novel views to gather features from underobserved regions. On Replica, it outperforms prior open-vocabulary methods, including LERF and OpenScene.

  • Problem

    Existing 3D segmentation methods rely on predefined classes, while explicit scene representations can limit resolution and alignment with pixel-based VLM features.

    Source: §1 · pp. 1–2

  • Method

    OpenNeRF distills pixel-aligned CLIP features into a NeRF and renders relevant novel views to extract additional open-set features.

    Source: §1 · p. 3

  • Results

    +4.5 mIoU over current state-of-the-art open-vocabulary methods is reported on the Replica dataset.

    Source: §1 · p. 3

  • Takeaways & Limitations

    NeRF-based representations with pixel-aligned VLM features are presented as effective for open-set 3D semantic segmentation and detecting small long-tail objects.

    Source: §1 · p. 3, §5 · p. 9

    Prior methods and comparisons remain constrained by limitations such as mesh resolution and poorly localized global CLIP features.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large visual-language models (VLMs), like CLIP, enable open-set image segmentation to segment arbitrary concepts from an image in a zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from a pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-set 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representations do not align well with the image-based nature of the visual-language models. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well to the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges, we propose OpenNeRF which naturally operates on posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in an overall less complex architecture without the need for additional DINO regularization. Our OpenNeRF further leverages NeRF's ability to render novel views and extract open-set VLM features from areas that are not well observed in the initial posed images. For 3D point cloud segmentation on the Replica dataset, OpenNeRF outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.

1 INTRODUCTION

OpenNeRF addresses open-set 3D segmentation by distilling pixel-aligned VLM features into a NeRF and using novel-view synthesis to inspect underobserved scene regions. It evaluates this setting against prior methods and reports improved segmentation on Replica.

1 INTRODUCTION

  • Open-set 3D segmentation queries arbitrary concepts rather than a predefined class set, supporting fine-grained scene understanding when labels are scarce or unavailable.The paper motivates this capability for previously unseen environments, robotics, and AR/VR applications.Source: §1 · pp. 1–2
  • Existing methods are constrained by mesh resolution or poorly localized global CLIP features, while OpenNeRF uses NeRF’s continuous representation to address these limitations.The paper also reports that NeRF-based representations better detect small long-tail objects than mesh-based representations.Source: §1 · pp. 2–3
  • OpenNeRF distills pixel-aligned CLIP features into a NeRF, avoiding LERF’s multi-resolution patches and additional DINO regularization.The design aims to improve localization and simplify the architecture relative to LERF.Source: §1 · pp. 2–3
  • Novel-view synthesis supplies additional visual-language features from relevant camera poses, focusing on scene regions missed by the original trajectory.The method identifies regions for further attention using disagreement across multiple views.Source: Figure 2 · p. 2, §1 · p. 3
  • The paper introduces an evaluation protocol comparing explicit mesh or point-cloud methods with implicit NeRF-based methods for open-set 3D semantic segmentation.This establishes a common evaluation setting for the task.Source: §1 · p. 3
  • +4.5 mIoU over current state-of-the-art open-vocabulary methods on Replica is reported for OpenNeRF.The paper presents this as its main quantitative result for open-vocabulary 3D segmentation.Source: §1 · p. 3

2 RELATED WORK

Related work spans pixel-level visual-language features, NeRF-based semantic fusion, and methods that lift 2D foundation-model information into 3D. OpenNeRF combines these directions for open-set 3D semantic segmentation and adds novel-view feature extraction.

2 RELATED WORK

  • OpenSeg provides class-agnostic pixel-aligned features that OpenNeRF distills into its NeRF representation for open-set segmentation.The paper uses OpenSeg features to retain pixel-level localization and support comparison with OpenScene.Source: §2 · p. 3
  • NeRFs provide a flexible mechanism for fusing 2D semantic information into 3D and rendering response maps or novel views for queried concepts.Prior work includes semantic fusion, localized text-prompt edits, feature distillation, and LERF’s multi-scale CLIP fields.Source: §2 · pp. 3–4
  • Prior open-set 3D methods lift 2D foundation-model features into coarse or point-cloud representations, with OpenScene identified as the closest existing point-cloud segmentation method.OpenNeRF differs by using a NeRF representation and rendering novel views of interesting scene parts.Source: §2 · p. 4

3 METHOD

OpenNeRF represents open-set 3D scenes as continuous NeRFs with pixel-aligned visual-language features, trains them with appearance, optional depth, and open-set objectives, and improves sparse or uncertain regions through uncertainty-guided novel-view rendering.

3 METHOD

  • OpenNeRF represents posed RGB images and pixel-aligned open-set features as a continuous volumetric scene that supports arbitrary-concept queries and zero-shot semantic assignment.Queries use normalized cosine similarity between encoded concepts and rendered open-set features.Source: §3 · p. 4, §3.1 · p. 4

3.1 OPEN-SET RADIANCE FIELDS

  • The open-set radiance field adds a D-dimensional feature to each 3D point and renders color, density, and features through volumetric integration.An additional MLP head models the open-set field alongside Mip-NeRF appearance and density.Source: §3.1 · p. 4

3.2 TRAINING OBJECTIVES

  • Training combines RGB appearance, optional depth supervision, and OpenSeg feature supervision, while excluding image-border rays and preventing inconsistent feature gradients from reaching density.The total objective is L = LRGB+λopen·Lopen+λdepth·Ldepth.Source: §3.2 · p. 5

3.3 RENDERING NOVEL VIEWS

  • 0.653 mean correlation links per-point uncertainty to open-set feature error across Replica scenes, supporting uncertainty-guided novel-view selection.High-uncertainty points are selected as targets for nearby camera poses, with NeRF density checks used to ensure visibility and avoid collisions.Source: §3.3 · p. 5, Figure 3 · p. 6

3.4 IMPLEMENTATION AND TRAINING DETAILS

  • The implementation uses Mip-NeRF in Jax, 3000 training iterations, 640 × 360 × 768 OpenSeg feature maps, float16 storage, and a pretrained CLIP text encoder.Open-set features are computed with OpenSeg and queries use the ViT-144@336 CLIP text encoder.Source: §3.4 · p. 6

4 EXPERIMENTS

OpenNeRF is evaluated on Replica using long-tail semantic labels and compared with OpenScene and LERF. It improves segmentation through NeRF-based feature representation, rendering choices, depth supervision, novel views, and open-set qualitative queries.

4 EXPERIMENTS

  • Replica provides 51 annotated semantic categories split into equal head, common, and tail subsets, including small objects and long-tail classes.The evaluation uses eight scenes and 200 posed RGB-D frames per scene after subsampling the original sequences.Source: §4 · p. 6, Figure 4 · p. 6

4.1 METHODS IN COMPARISON.

  • OpenNeRF is compared with OpenScene’s distilled and ensemble models and with LERF using CLIP- and OpenSeg-based open-set features.OpenScene additionally benefits from large-scale 3D and Matterport pre-training, unlike LERF and OpenNeRF.Source: §4.1 · p. 7, §4.2 · p. 7, Table 1 · p. 7

4.2 RESULTS ON 3D SEMANTIC SEGMENTATION

  • +4.5 mIoU over OpenScene across all classes, with gains of +3.7 head, +5.6 common, and +4.3 tail.OpenNeRF also detects categories that OpenScene misses, although many long-tail classes remain undetected by both methods.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 ANALYSIS EXPERIMENTS

  • Projection of rendered features onto the point cloud outperforms direct NeRF feature sampling, while direct sampling already exceeds OpenScene and LERF.The projection strategy can accumulate multiple samples along each ray through volumetric rendering.Source: §4.3 · p. 9, Table 2 · p. 9
  • Depth supervision further improves the open-set feature field by adding a Huber loss between rendered distance and ground-truth depth.The authors attribute this gain to improved volumetric reconstruction quality.Source: §4.3 · p. 9, Table 2 · p. 9
  • Rendered novel views improve performance, whereas random camera placement causes a drastic 15.4 mIoU drop when views enter geometry or lack meaningful context.Views along the original camera trajectory do not provide the same improvement, isolating the contribution of the proposed novel-view strategy.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 QUALITATIVE RESULTS FOR 3D SCENE SEGMENTATION AND OPEN-SET APPLICATIONS

  • OpenNeRF produces less noisy masks than OpenScene and correctly segments examples such as wall-plugs, blankets, baskets, and blinds.Its open-set queries also cover object properties and materials, while in-the-wild iPhone scans demonstrate application to unseen scenes.Source: Figure 5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5 CONCLUSION

OpenNeRF represents open-set 3D scenes with a NeRF and pixel-aligned VLM features, using novel-view synthesis to obtain additional features. It outperforms OpenScene and LERF on unsupervised 3D semantic segmentation.

  • OpenNeRF is a NeRF-based representation for open-set 3D semantic scene understanding that combines pixel-aligned VLM features with novel-view synthesis.The method identifies regions where additional novel views should be generated.Source: §5 · p. 9
  • OpenNeRF outperforms both the mesh-based OpenScene and the NeRF-based LERF in experiments.Source: §5 · p. 9
Loading 2404.03650v1…