Source-linked AI summary
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, Phillip Isola
TL;DR
Robotic manipulation needs both rich semantic understanding and detailed 3D geometry for open-set generalization. This work uses distilled feature fields to combine 2D foundation-model priors with 3D geometry, enabling few-shot 6-DOF manipulation and language-guided generalization to unseen objects and categories.
Problem
Robotic manipulation requires semantic and geometric understanding to identify target objects and determine stable grasps in cluttered, unpredictable scenes.
Method
The method distills dense visual or vision-language features into NeRF-based feature fields, using hierarchical hashgrids and MaskCLIP to support efficient 3D modeling and language guidance.
Results
The robot generalized in few-shot grasping and placing to unseen object shapes, appearances, materials, poses, and categories, including free-text language-guided manipulation.
Takeaways & Limitations
Distilled feature fields combine 2D visual or language priors with 3D geometry for open-ended scene understanding and manipulation without fine-tuning.
Takeaways & Limitations
The system requires 1m 40s to collect 50 scene images and 90s to model the NeRF and feature field, limiting suitability for dynamic manipulation.
Abstract
from arXiv · showhide
Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects.
1 Introduction
The paper combines 3D scene geometry with semantic priors from pretrained image models for few-shot, language-guided manipulation of novel objects. Its F3RM system distills dense visual features into 3D feature fields and supports open-set generalization in grasping and placing.
- Robotic manipulation requires both geometric understanding for stable grasps and semantic understanding to identify requested objects amid clutter.
- F3RM constructs a Distilled Feature Field by rendering pretrained image features alongside RGB colors in a NeRF-based 3D scene representation.
- The work studies DFFs for robotic manipulation using self-supervised DINO features and vision-language features from CLIP.
- Hierarchical hashgrids reduce scene-modeling time, while MaskCLIP supplies dense patch-level CLIP features aligned with language.
- F3RM supports few-shot grasping and placing across unseen object shapes, appearances, materials, poses, and categories, including free-text language commands.
2 Problem Formulation
The paper formulates manipulation as predicting a rigid-body grasp or placing pose from a few demonstrations or an open-text object description. Evaluation targets open-ended generalization to cluttered scenes containing previously unseen objects and categories.
- The task class consists of grasping and placing problems parameterized by a single rigid-body transformation T ∈SE(3).
- Few-Shot Manipulation: In few-shot manipulation, demonstrations pair multiple RGB camera views with a task-achieving pose, and testing requires predicting a pose in a new cluttered scene.
- Few-Shot Manipulation: Open-ended few-shot evaluation varies object shape, size, pose, and material between demonstrations and test scenes.
- Open-Text Language-Guided Manipulation: Language-guided manipulation uses positive and negative text queries to select objects, including categories absent from the demonstrations.
3 Feature Fields for Robotic Manipulation (F3RM)
F3RM distills pretrained visual and language representations into 3D feature fields, then uses them to encode demonstrations, infer 6-DOF poses, and guide manipulation with open-text commands.
- Feature Field Distillation: F3RM distills pretrained vision and vision-language representations into 3D feature fields for open-ended robotic manipulation.The approach addresses feature-field construction, 6-DOF pose representation and inference, and language guidance.
- Feature Field Distillation: Dense 2D descriptors are rendered into a 3D field by extending NeRF with feature outputs supervised by pretrained vision-model feature maps.The field maps 3D positions to feature vectors and is trained through a feature-rendering integral between near and far planes.
- Feature Field Distillation: MaskCLIP extracts dense CLIP patch features while preserving alignment with language embeddings, enabling language-guided feature distillation.Position-encoding interpolation accommodates larger images with arbitrary aspect ratios.
- Representing 6-DOF Poses with Feature Fields: Demonstration poses are represented by alpha-weighted features at fixed canonical-gripper query points, concatenated into pose embeddings and averaged into a task embedding.Query points are manually distributed over target geometry, contextual cues, and free space; averaging demonstrations helps reject spurious features.
- Representing 6-DOF Poses with Feature Fields: Pose inference filters occupied, task-relevant voxels, samples rotations, optimizes similarity to the task embedding, rejects collisions, and sends ranked valid poses to a motion planner.The method operates over the entire feature field without requiring segmentation masks or object poses.
- Open-Text Language-Guided Manipulation: Language guidance retrieves demonstrations by CLIP similarity, then optimizes poses using a normalized similarity between the text embedding and alpha-weighted query-point features.The selected demonstrations need not contain objects from the same category as the target object.
4 Results
The experiments evaluate few-shot 6-DOF manipulation and open-text language-guided manipulation with distilled feature fields. DINO and CLIP provide complementary strengths, while language-guided execution generalizes to novel categories but remains unreliable.
- Few-Shot Grasping and Placing: Five 6-DOF grasping and placing tasks are evaluated with two demonstrations per task, comparing DINO ViT, CLIP ViT, CLIP ResNet, and NeRF-based baselines.The baselines use NeRF density, RGB color values, or intermediate NeRF features.
- Few-Shot Grasping and Placing: DINO and CLIP feature fields generalize better semantically than density, RGB, or intermediate NeRF features in cluttered scenes.The baselines struggle to identify the target semantic category when distractors are present.
- Few-Shot Grasping and Placing: DINO captures part-level geometry, whereas CLIP favors semantic and categorical information but is weaker on geometric features needed for grasping and placing.The reported failures attribute 7/19 DINO failures and 21/27 combined CLIP failures to grasp-rotation or translation inaccuracies.
- Language-Guided Manipulation: The language-guided evaluation uses 13 tabletop scenes containing novel same-category objects and out-of-distribution objects from new categories.The test scenes reuse ten demonstrations spanning four object categories.
- Language-Guided Manipulation: CLIP failures commonly arise from bag-of-words behavior, while six of 19 failures stem from rotational or translational grasp-prediction errors.The reported examples include confusion among black objects and misinterpreting relations such as “mug on a can of spam.”
- Language-Guided Manipulation: 31 out of 50 language queries succeed, including queries specifying object properties and spatial relations, with generalization to bowls, tape rolls, markers, and utensils.The authors state that this success rate is far from practical industrial use.
5 Related Work
The work relates to language-guided robotics, dense visual descriptors, geometric representations, and 3D feature fields. Its distinction is combining geometric information with pretrained visual semantics for robotic manipulation.
- Open-Ended Generalization via Language: Prior robotics work uses natural language for task specification, planning, and navigation, while F3RM instead incorporates geometric information with pretrained visual features.The related work contrasts this approach with jointly pretrained foundation models for robotics.
- Dense 2D Visual Descriptors: Dense descriptor methods use dynamic reconstruction, multi-view RGB-D scenes, or NeRF supervision to associate or learn visual features in 3D.These approaches address correspondence and reconstruction across pixels, views, thin structures, or reflective materials.
- Geometric Aware Representations for Robotics: Geometric-aware representations support mapping, grasping, and locomotion, using direct 3D supervision or representations learned from posed images and videos.Neural scene representations can handle reflective or transparent objects and fine geometry.
- 3D Feature Fields: 3D feature-field work integrates 2D foundation models with neural fields outside manipulation, whereas this work extracts dense patch-level CLIP features for robotic manipulation.The paper distinguishes its features from LERF’s multi-scale image-level features.
6 Conclusion
The paper combines 2D visual priors with 3D geometry for open-ended few-shot and language-guided manipulation. It reports broad out-of-the-box generalization, but scene construction remains slow and limits dynamic use.
- Conclusion: Distilled Feature Fields combine 2D visual priors with 3D geometry for open-ended scene understanding in few-shot and language-guided manipulation.The conclusion frames this combination as the paper’s central approach.
- Conclusion: Without fine-tuning, the approach generalizes across object categories, materials, and poses.The conclusion describes this as out-of-the-box generalization.
- Conclusion: Vision-language features provide language guidance at multiple levels of semantic granularity.This capability is stated as a property of distilled feature fields sourced from vision-language models.
- Limitations: The system takes 1m 40s to collect 50 scene images and 90s to model the NeRF and feature field.The authors identify faster geometry recovery from few views as necessary for closed-loop dynamic manipulation.
A.1 Neural Radiance Fields (NeRFs)
NeRF represents scenes as continuous functions of position and viewing direction, then renders images by integrating emitted color along camera rays. Training optimizes density and color against observed RGB images.
- NeRF Representation: A NeRF maps position x and normalized viewing direction d to differential density σ and emitted RGB color.The density network depends only on position, while the color network depends on position and viewing direction.
- Novel-View Synthesis: NeRF synthesizes each image pixel by casting a ray from the camera through the pixel and parameterizing points as r_t = o + td.The ray is evaluated between near and far scene bounds.
- Novel-View Synthesis: The rendered ray color is computed with a volume-rendering integral weighted by density and accumulated transmittance.T(t) denotes accumulated transmittance along the ray from the near bound to the sampled point.
- Modeling a Scene with NeRFs: Given N RGB images, NeRF optimizes density σ and color c by minimizing photometric error between observed pixel colors and discretely rendered colors.The discrete rendering approximation is used to estimate the color for each ray.
A.2 Dense 2D Feature Extraction via MaskCLIP
MaskCLIP extracts dense features from CLIP by modifying its computation graph to return patch-level features rather than only the CLS token.
- The original CLIP image-feature algorithm returns only the CLS token.
A.3 Feature Fields
The feature-field design distills 2D visual features into a 3D representation while preserving fine spatial detail and reducing feature-reconstruction error with hash grids.
- Feature-field training reconstructs patch-level feature maps without upscaling them to pixel resolution, reducing memory demands.
- Feature fields recover higher 3D detail than the low-resolution source 2D feature maps.
- The hash grid architecture consistently achieves lower feature-distillation error than the MLP-head architecture.
A.3.1 Ablation on Feature Field Architecture
The architecture ablation compares a hierarchical hash grid with an MLP head and finds that hash grids better preserve high-frequency detail and semantic object boundaries.
- The hierarchical hash grid takes 3D position as input and outputs a feature vector, unlike the compared MLP head that uses intermediate NeRF features.
- The hash grid achieves lower MSE because it captures higher-frequency signals and finer details.
- Hash-grid feature fields produce more well-defined semantic boundaries between objects despite only a marginal MSE difference.
A.4 Experimental Setup
The experiments evaluate feature-field manipulation across physical setup, modeling, query sampling, pose optimization, baselines, and view-count ablations. They show practical modeling constraints, baseline-specific failures, and a lower bound of roughly 20 training views before quality declines sharply.
- Physical Setup: The robot scans tabletop scenes with 50 RGB images from three helical passes using a selfie-stick-mounted camera.
- NeRF and Feature Field Modeling: NeRF and feature fields are optimized sequentially for 2000 steps each, taking at most 90 seconds on an NVIDIA RTX 3090.
- NeRF and Feature Field Modeling: Feature extraction uses model-specific original feature-map resolutions rather than RGB-image resolution.
- Ablation on Number of Training Views: Reducing training views increases floaters, with approximately 20 images serving as the lower bound before a drastic quality decline.
- Grasp Pose Optimization: The pose pipeline samples 100 query points, filters free space and low feature similarity, then optimizes candidate poses before motion planning.
- Baselines: MIRA requires manual rotation selection and filtering of floater or unreasonable grasps, while its 2.5D formulation struggles with occlusions and distractors.
- Baselines: DINO can overfit to color and distractors, whereas CLIP struggles less with distractors because of stronger semantic understanding.
- Language-Guided Manipulation: Language-guided grasp-pose inference takes 6.9 seconds on average and commonly converges within 50–100 optimization steps.