Source-linked AI summary
NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, Daniel Duckworth
TL;DR
NeSF addresses the challenge of producing 3D semantic segmentation without costly 3D sensors or scarce 3D annotations. It learns a semantic field from posed RGB images and 2D semantic maps over implicit density fields, achieving comparable baseline quality while supporting novel-view rendering and direct 3D queries. Its accuracy remains constrained in challenging settings and by limited appearance and fine-grained geometric information.
Problem
3D semantic segmentation often requires costly 3D sensors, scarce 3D annotations, or externally supplied 3D geometry, whereas 2D methods overlook scene structure.
Method
NeSF applies a neural network to an implicit NeRF density field and trains the resulting semantic field using posed 2D semantic maps rendered through volumetric rendering.
Results
NeSF performs comparably to competitive baselines on simpler scenes, matches DeepLab on novel camera poses, and reaches near 97% 3D segmentation accuracy with as few as 25 RGB images for NeRF optimization.
Takeaways & Limitations
NeSF provides dense 3D semantic fields that can be queried directly or rendered from novel camera poses using only posed 2D information at training and test time.
Takeaways & Limitations
NeSF underperforms baselines in more challenging settings and can smear labels across nearby objects without appearance or fine-grained geometry.
Abstract
from arXiv · showhide
We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in implicit neural scene representations wherein 3D structure is captured by point-wise functions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be queried at arbitrary 3D points. Notably, NeSF is compatible with any method producing a density field, and its accuracy improves as the quality of the density field improves. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realistically rendered synthetic scenes. Our method is the first to offer truly dense 3D scene segmentations requiring only 2D supervision for training, and does not require any semantic input for inference on novel scenes. We encourage the readers to visit the project website.
1. Introduction
NeSF addresses the scarcity of practical 3D semantic supervision by learning dense 3D semantic fields from posed RGB images and 2D semantic maps. It uses implicit scene geometry to support novel-view rendering, direct 3D queries, and generalization to unseen scenes.
- 3D semantic understanding supports applications including visual navigation and robotic interaction.
- Existing 2D methods produce per-pixel annotations while largely ignoring the scene’s underlying 3D structure.
- 3D segmentation methods commonly require expensive 3D sensors, scarce 3D annotations, or an input classical 3D representation.
- NeSF maps 3D points to semantic-category probability distributions using an implicit geometry representation and volumetric rendering for image-space supervision.
- NeSF produces dense 2D and 3D segmentations for novel scenes from posed RGB images and semantic maps, with as few as one semantic map per scene during training.
- The accompanying datasets contain over 1,000 scenes and 3,000,000 realistically rendered, semantically annotated frames for 2D and 3D evaluation.
2. Related works
Prior work largely treats semantic segmentation as a fully supervised single-modality task or combines image supervision with an externally supplied 3D substrate. NeSF instead reconstructs and segments dense 3D representations from 2D inputs and supervision alone, targeting novel scenes without test-time semantic supervision.
- Conventional semantic segmentation methods target fully supervised 2D or 3D inputs represented as images, point clouds, voxel grids, or meshes.
- Hybrid multiview methods use image supervision but still require an input 3D substrate and reason primarily in the image domain.
- NeSF reconstructs and segments a dense 3D representation from 2D inputs and supervision without ground-truth 3D annotations or input geometry.
- Atlas requires posed photos at test time but still relies on 3D supervision during training.
- Unlike most NeRF-based semantic methods, NeSF generates 3D semantic segmentations for novel scenes without supervision at test time.
3. Method
NeSF converts per-scene NeRF density fields into 3D semantic fields using shared spatial and feature-processing networks, trained from rendered 2D semantic maps. The method combines density extraction, 3D reasoning, feature decoding, volumetric rendering, and augmentation.
- NeRF pre-training: NeRF models are pretrained independently from posed RGB images, producing per-scene volumetric density fields used as implicit geometry.The radiance component is discarded after fitting, leaving density fields for semantic processing.
- Supervision: Volumetric rendering converts the predicted 3D semantic field into 2D maps, which are supervised against ground-truth semantic maps with cross-entropy and smoothness regularization.The regularizer encourages similar predictions in local neighborhoods.
- Semantic reasoning: NeSF uniformly samples each density field onto a 3D lattice, then applies a 3D UNet to produce a spatial feature grid.The UNet enables local-neighborhood reasoning because a point-wise density value alone does not capture 3D structure.
- Semantic reasoning: Shared translation-network parameters allow the semantic model to generalize across scenes and support novel-scene inference.The shared network transforms density-derived features into semantic representations rather than fitting separate semantic parameters per scene.
- Semantic reasoning: A decoder interpolates features at arbitrary 3D query points and outputs probability distributions over semantic categories.Trilinear interpolation supplies the feature vector queried by the multilayer perceptron decoder.
- Data augmentation: Random rotations around the z-axis augment density grids without requiring NeRF retraining.The augmentation queries the existing NeRF at rotated coordinates before semantic processing.
4. Datasets – Table 1 and Figure 3
The paper introduces three synthetic multiview datasets with hundreds of randomly composed scenes and controlled rendering conditions. Their camera and scene partitions support both novel-view and novel-scene evaluation.
- Dataset construction: KLEVR, ToyBox5, and ToyBox13 together provide hundreds of photorealistically rendered synthetic scenes for multiview reconstruction and semantic segmentation.The datasets were created with Kubric to eliminate distractors such as motion blur, calibration error, and object motion.
- KLEVR: KLEVR contains 4–12 simple geometric objects whose geometry defines semantic classes while color, scale, and placement act as distractors.Each scene includes 300 frames from randomly sampled camera poses.
- ToyBox5 and ToyBox13: ToyBox5 and ToyBox13 use ShapeNet objects and HDRI backdrops, with objects sampled from five and thirteen common categories, respectively.Each scene also contains 4–12 randomly placed objects and 300 independently sampled camera poses.
- Dataset examples: Each dataset frame contains an RGB image, semantic map, and depth map for multiview supervision and evaluation.The depth map is referenced in the dataset examples but not pictured in Figure 3.
- Evaluation splits: Scenes are split into TRAIN SCENES and NOVEL SCENES, while each scene’s frames are divided into TRAIN CAMERAS and TEST CAMERAS.This design supports evaluation both from novel views within observed scenes and across previously unseen scenes.
5. Experiments
NeSF is evaluated against 2D and 3D baselines across three datasets, then analyzed through ablations of architecture, reconstruction quality, and supervision density. It performs competitively in several settings but loses accuracy on harder datasets and when compared with fully supervised 3D input.
- Comparisons to baselines: NeSF is evaluated on KLEVR, ToyBox-5, and ToyBox-13 against DeepLab for 2D segmentation and SparseConvNet for 3D segmentation.The evaluation uses posed RGB images and semantic maps for NeSF and DeepLab, while SparseConvNet receives oracle 3D point clouds and 3D supervision.
- Comparisons to baselines: On ToyBox13, NeSF underperforms DeepLab by 6.6% in 2D mIoU but achieves near-identical accuracy on novel camera poses that DeepLab cannot approach.NeSF is evaluated on additional camera poses where RGB information is unavailable.
- Comparisons to baselines: NeSF underperforms SparseConvNet by 4.7-5.2% on ToyBox5 and 19.8-23.1% on ToyBox13.SparseConvNet has dense ground-truth depth maps and full 3D supervision, giving it an upper-bound advantage over NeSF.
- Comparisons to baselines: NeSF separates similarly appearing objects with different geometry but struggles with thin structures and smears labels across nearby objects.The qualitative comparison attributes these errors to limited spatial resolution and the absence of appearance or fine-grained geometry.
- Ablation Studies: Random scene rotations improve 2D and 3D mIoU by 10.3% and 11.8%, while higher density-grid resolution and model capacity also improve segmentation quality.The ablation evaluates 25 KLEVR scenes and identifies data augmentation as the most influential varied component.
- Ablation Studies: NeSF accuracy improves monotonically with NeRF reconstruction quality, while 3D segmentation reaches about 97% with as few as 25 RGB images and generalizes with one semantic map per training scene.Additional semantic maps help mainly with small numbers of scenes and have no noticeable effect after 25 scenes.
6. Conclusions and Limitations
NeSF generates dense 3D semantic fields from posed 2D images and supports direct 3D queries or rendering from novel camera poses. The method is evaluated with new datasets, but underperforms baselines in more challenging settings.
- NeSF jointly reconstructs and segments 3D scenes using only posed 2D RGB images and semantic maps.
- The resulting dense semantic field can be queried directly in 3D or rendered as 2D semantic maps from novel camera poses.
- NeSF fuses information across independent views while operating on posed 2D information alone during training and testing.
- NeSF underperforms its baselines in more challenging settings.
- The authors introduce three datasets for multiview reconstruction and semantic segmentation totaling over 3,000,000 frames and 1,000 scenes.
- The datasets include realistic illumination, diverse objects and backgrounds, and accompanying code and pretrained NeRF models planned for public release.
Supplementary Material
The supplementary material includes an overview video and author-contribution descriptions covering implementation, experiments, datasets, and project oversight.
- The paper recommends viewing a high-definition overview video.
- Contributions also include experiment suggestions, codebase development, scalable dataset infrastructure, and research or technical leadership.
- The author-contribution section assigns responsibilities for NeSF implementation, experiments, dataset generation, baselines, and software development.
B. Training Details
Training uses posed images from training scenes and evaluates methods on novel scenes, with separate NeRF, NeSF, DeepLab, and SparseConvNet configurations.
- Unless otherwise stated, methods train on all training scenes using 9 randomly selected images per scene and evaluate on 4 images from novel scenes.
- NeRF provides each scene’s density field through a per-scene model trained from randomly sampled pixels across 9 views.
- NeSF uses a 3D UNet followed by an MLP decoder as its two major model components.
- NeSF training uses Adam with an exponentially decaying learning rate from 1e−3 to 1e−5 over 25,000 steps.
- Each NeSF training step samples scenes, pixels, and 192 points along each ray for volumetric rendering.
- Training adds random scene rotations and noisy 3D coordinates for smoothness regularization, weighted at 0.1.
- NeSF reaches convergence in approximately 45 minutes on 32 TPUv3 cores.
- DeepLab uses a COCO-pretrained Wide-ResNet-38 optimized with SGD and momentum on 32 TPUv3 chips.
C.1. Qualitative Results
Qualitative results show that NeSF generally separates foreground objects from floors and backgrounds, but its 3D density fields miss thin structures and fine details.
- Across qualitative results, all methods effectively separate foreground objects from the floor and background.
- NeSF and DeepLab may assign different semantic categories to parts of the same object when the correct category is ambiguous.
- NeSF and SparseConvNet remain multi-view consistent, whereas DeepLab can change its classification between views.
- NeSF density fields often miss thin structures and fine details, with floaters especially evident in ToyBox13.
C.2. Ablations
NeSF remains effective with sparse semantic supervision and improves as scene reconstruction quality increases, while producing consistent predictions across views. Its performance is also sensitive to density-grid and model-design choices, and floaters remain a failure source.
- Model ablations: Random scene rotations, increased density-grid resolution, and greater UNet capacity improve both 2D and 3D mIoU.Random scene rotations produced the largest ablation gains on ToyBox5, adding 9.3% to 2D mIoU and 6.1% to 3D mIoU.
- Multiview consistency: NeSF and SparseConvNet preserve multiview classifications, whereas DeepLab’s predictions vary across independent views.
- Reconstruction quality: NeSF’s accuracy improves monotonically with NeRF reconstruction quality as more RGB images are used to fit the density field.This sensitivity is reported for both 2D and 3D segmentation on ToyBox5.
- Multiview consistency: NeSF’s epipolar-plane predictions are smooth and consistent except when a floater obstructs the camera.The same density-field artifact can produce mislabeled semantic pixels across video frames.
- Data efficiency: NeSF generalizes to new scenes with as little as one semantic map per scene, while additional maps provide only marginal improvement.On ToyBox5, performance improves with more maps, with a large jump between 5 and 10 and saturation around 25 maps per scene.
C.4. Limitations
NeSF’s main limitation is its dependence on the quality of the recovered density field: floaters can obscure foreground objects and create semantic errors, especially in rendered 2D views. Errors also concentrate on small, thin, or geometrically similar object categories.
- Category errors: On ToyBox13, NeSF struggles with small or geometrically unarticulated objects, including rifles and loudspeakers.Reported accuracies are 56.3% 2D and 75.3% 3D for rifles, and 38.5% 2D and 40.4% 3D for loudspeakers.
- Category errors: Foreground confusions commonly occur between geometrically similar classes, such as benches versus chairs or sofas and loudspeakers versus tables.These confusions appear in both 2D and 3D semantic classifications.
- Geometric artifacts: Floaters can obstruct the camera, obscure correctly labeled foreground objects, and lower 2D mIoU relative to 3D mIoU.The paper identifies improved geometric reconstruction as the route to reducing this failure.