Source-linked AI summary
Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations
Vadim Tschernezki, Iro Laina, Diane Larlus, Andrea Vedaldi
TL;DR
Existing scene-specific 3D reconstruction methods often capture appearance and geometry without high-level semantics. N3F distills 2D image features into a 3D student through neural rendering, consistently improving feature-based tasks including retrieval, segmentation, and editing across static and dynamic scenes.
Problem
Scene-specific 3D reconstruction methods often produce low-level appearance and geometry representations that do not capture high-level semantics.
Method
N3F distills a 2D feature teacher into a 3D student neural feature field trained alongside radiance through differentiable rendering.
Results
N3F consistently improves 2D feature baselines and supports object retrieval, segmentation, and scene editing across static and dynamic scenes.
Takeaways & Limitations
The fused representation provides scene-specific semantic understanding without manual labels and enables 3D object understanding beyond the original 2D teacher.
Takeaways & Limitations
N3F inherits source-feature limitations, including DINO's tendency to group semantically related objects such as fruits, vegetables, and utensil handles.
Abstract
from arXiv · showhide
We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels, but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark.
1. Introduction
N3F integrates 2D semantic features with 3D reconstruction by distilling image-space features into a neural field. This improves scene-specific feature consistency and supports retrieval, segmentation, and editing across static and dynamic scenes.
- N3F maps 2D semantic features into equivalent 3D features using neural rendering and backpropagation.The approach uses the same rendering equations as view synthesis to train the 3D representation.
- N3F distills a 2D teacher network into a 3D student, improving viewpoint independence, occlusion awareness, and feature consistency.The resulting student can surpass the teacher for understanding a particular scene.
- Self-supervised features provide an open-world starting point for recognizing objects, parts, and correspondences without predefined labels or annotations.This setting is especially relevant to user-specific and evolving egocentric video scenarios.
- The method is evaluated on static scenes with vanilla NeRF and on dynamic egocentric videos from EPIC-KITCHENS.The latter includes long first-person cooking sequences with static and dynamic components.
- N3F systematically improves object retrieval and is also demonstrated for 3D object segmentation, amodal segmentation, and scene editing.The retrieval evaluation tests whether an object queried in one frame can be found despite viewpoint changes or occlusions.
2. Related Work
Related work spans self-supervised feature extraction, neural rendering for static and dynamic scenes, and semantic or object-centric radiance fields. N3F differs by distilling broadly applicable image features into 3D without relying on semantic or 3D labels.
- Self-supervised visual features: Self-supervised visual features support unlabeled object localization, semantic and part segmentation, and point correspondences.DINO is highlighted for learning localized representations that can segment salient objects without labels.
- Neural rendering: Neural rendering represents geometry and radiance with implicit functions fitted to images through differentiable rendering, while extensions address dynamic scenes.Dynamic-scene approaches include time-dependent flow fields, canonical models, and space warping.
- Semantic and object-centric neural rendering: Semantic and object-centric neural rendering augments radiance fields with scene decomposition, object compositions, or semantic predictions.Existing methods include semantic fusion from 2D labels, joint cross-scene inference, and object-aware modeling for dynamic scenes.
- Semantic and object-centric neural rendering: Unlike related semantic-rendering methods that use semantic or 3D labels, N3F builds on self-supervised features for static and dynamic 3D-consistent segmentation.
- Feature fusion: Concurrent related work fuses features similarly but emphasizes applications involving multiple modalities, segmentation queries, and scene editing.
- Feature distillation: Feature distillation transfers knowledge from complex teacher models into smaller students and can sometimes improve student performance over the teacher.
3. Method
N3F distills 2D image features into a 3D neural field alongside scene reconstruction, using differentiable rendering to support view-consistent feature representations and downstream 2D and 3D applications.
- 3.1. Neural Feature Fusion Fields: Known camera projections and ray samples connect images to 3D points, allowing volumetric rendering and backpropagation to transfer features between image and scene representations.The underlying neural radiance field maps 3D points to occupancy and color and is learned from images and camera poses.
- 3.1. Neural Feature Fusion Fields: N3F generalizes neural rendering by reconstructing feature images alongside color images, using a neural feature field with shared occupancy and feature outputs.The feature field is represented by (σ, cΦ), while RGB rendering uses (σ, crgb).
- 3.1. Neural Feature Fusion Fields: The method treats the image-space feature extractor as a 2D teacher and the world-space feature network as a 3D student.The student is trained with image and feature reconstruction losses, combined using λ, while teacher features provide the distillation target.
- 3.2. Advanced Neural Rendering Models: N3F can be implemented with multiple neural rendering variants, including vanilla NeRF for static scenes and NeuralDiff for complex egocentric videos.The dynamic-video setting contains static background objects, manipulated foreground objects, and the actor’s body parts.
- 3.3. Applications of N3F: The distilled representation supports 2D object retrieval, 3D object segmentation, scene editing, and amodal segmentation from image-region queries.For 3D segmentation, features are used directly in 3D before rendering; scene editing suppresses selected points by setting their occupancy to zero.
4. Experiments
N3F is evaluated on static LLFF scenes and dynamic EPIC-KITCHENS videos for retrieval, segmentation, amodal segmentation, and scene editing. Across these tasks, distilled 3D features improve retrieval and enable 3D-aware operations from 2D queries, while inheriting limitations of the source features.
- Experimental setup: Experiments use vanilla NeRF for static LLFF scenes and NeuralDiff for dynamic EPIC-KITCHENS cooking videos.Pre-trained models are continued for feature distillation, with the feature prediction head initially isolated during training.
- Experimental setup: The evaluation covers 2D object retrieval, 3D object segmentation, amodal segmentation, and scene editing.Quantitative evaluation uses mean average precision for annotated 2D object retrieval, while the other tasks are illustrated qualitatively.
- 2D object retrieval: 11.9 mAP is the smallest absolute gain from distilling DINO features, while DeiT improves from 47.5 to 74.5 mAP.The reported improvements occur across all feature extractors and scenes, with the largest gap for the supervised DeiT features.
- 2D object retrieval: N3F produces sharper, more confident retrieval maps under viewpoint and reflectance changes, and retrieves more complete objects than 2D DINO features.The grater example highlights robustness to metallic reflections, while LLFF results show more detailed and complete segmentations from small patches.
- 3D segmentation and scene editing: N3F extracts queried object geometry in 3D without 3D training labels and supports scene editing by suppressing matched 3D occupancies.Radiance-field scene knowledge can produce valid backgrounds after object removal, particularly in dynamic scenes where backgrounds were observed at other times.
- Amodal segmentation: N3F retrieves occluded object parts that 2D DINO cannot segment because the teacher is limited to 2D representations.Its 3D awareness allows segmentation of objects such as a cutting board even when barely visible behind an actor’s arm.
- Limitations: N3F inherits source-feature limitations, including grouping semantically related fruits, vegetables, and utensil handles in EPIC-KITCHENS.This limitation is specifically observed for self-supervised features such as DINO.
5. Conclusions
N3F boosts the 3D consistency of 2D image features across reconstructible image sets and benefits object retrieval, segmentation, and editing in static and dynamic scenes.
- N3F improves the 3D consistency of 2D image features within image sets reconstructible through neural rendering.
- The method works with multiple neural rendering models across static objects and challenging egocentric videos containing dynamic scenes.
- Experiments demonstrate benefits for object retrieval, segmentation, and scene editing.
- Future work includes integrating N3F into self-supervised feature learning and fusing videos for cross-instance correspondences.
A. Neural Feature Fusion Fields with CNNs
N3F also distills CNN features effectively, substantially improving object-retrieval performance over the teacher features on EPIC-KITCHENS scenes.
- 48.9 mAP with NeuralDiff-N3F improves by about 20 mAP over 29.7 mAP for DeepLab teacher features.
- The evaluation uses DeepLab CNN features with self-supervised MoCo-v3 weights across 10 EPIC-KITCHENS scenes.
- The results show that N3F is applicable beyond vision-transformer features to convolutional neural network features.
B. Foreground segmentation in videos
The video segmentation evaluation targets moving objects and actors, with NeuralDiff-N3F producing higher average precision than NeuralDiff, especially for infrequently moved objects.
- NeuralDiff-N3F segments dynamic objects with higher average precision than the original NeuralDiff method.
- The precision advantage is especially evident for objects moved less frequently, including cutting boards in both illustrated rows.
- The segmentation objective covers every object moved during the video sequence and additionally includes the actor.
- Training uses all images because the evaluation focuses on foreground segmentation rather than photometric reconstruction.
C. Data and protocol for NeuralDiff-N3F
The EPIC-KITCHENS protocol annotates five objects per scene and uses expanded higher-resolution reconstructions, with rules addressing manipulation, visibility, and whole-object labeling.
- Each scene receives annotations for 5 objects, with gallery subsets divided into validation and testing sets.
- The dataset extends prior reconstructions by downscaling full HD images by 4 instead of 8 for more detailed rendering.
- EPIC-Diff uses about 800 frames per scene, although scenes can contain up to 90000 total frames.
- The retrieval protocol renders 3D features for an unseen-view query patch and retrieves pixels with similar features.
- Annotation rules focus on manipulated objects and mark only each object's visible part in every frame.
D. Additional qualitative results
Additional LLFF scenes show that queried-feature retrieval remains neighborhood-biased, while fortress features are less noisy and PCA-reduced features are more discriminative by RGB intensity.
- The t-rex scene retains a bias toward the neighborhood surrounding the queried patch.
- The fortress scene exhibits less noise in distances between queried and retrieved features.
- PCA-reduced features show higher discriminative power when their three dimensions are visualized as RGB channels.