Source-linked AI summary
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, Vincent Sitzmann
TL;DR
The paper addresses few-shot manipulation of unseen same-category objects across arbitrary 3D poses, where successful actions require both spatial contact and orientation. It introduces Neural Descriptor Fields, which learn category-level 3D descriptors and recover demonstrated poses by optimization; across three pick-and-place tasks, the system exceeds 85% overall success with 10 demonstrations and outperforms 2D baselines.
Problem
The goal is to learn manipulation of unseen same-category objects into target configurations regardless of their 3D location and orientation, using only a few demonstrations.
Method
Neural Descriptor Fields encode object points and relative object-target poses with category-level descriptors, then jointly recover coordinate-frame poses by optimizing descriptor agreement using SE(3)-equivariant 3D representations.
Results
Above 85% overall success was achieved on three pick-and-place tasks for unseen object instances in out-of-distribution configurations using 10 expert demonstrations, consistently outperforming 2D non-SE(3)-equivariant baselines.
Takeaways & Limitations
NDFs support few-shot manipulation transfer across object instances and novel rotations and translations without expert-labeled keypoints or further imitation-time training.
Takeaways & Limitations
Non-rigid-object applicability remains untested, trajectory transfer requires future integration with trajectory optimization, and the placement target is assumed static.
Abstract
from arXiv · showhide
We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/.
I. INTRODUCTION
NDFs address few-shot manipulation across object instances and arbitrary 6-DoF poses by learning task-relevant geometric correspondences without labeled keypoints. They encode descriptors in continuous 3D fields and recover local frames by SE(3)-equivariant pose matching.
- Motivation: Current demonstration-learning methods require many examples spanning object instances, positions, and orientations to manipulate novel objects reliably.This motivates data-efficient transfer across object categories and 6-DoF configurations.
- Task Representation: Task-relevant local frames differ by manipulation, such as rim grasping or aligning a rack with a mug’s handle or bottom surface.The relevant geometric structure depends on whether the task is grasping or placing.
- Neural Descriptor Fields: NDFs represent each object point cloud as a continuous function mapping 3D coordinates to descriptors consistent across category-level geometric features.The queried coordinates need not lie on the object and may be occluded.
- Pose Matching: Rigid query points encode local coordinate frames, while optimizing their SE(3) transformation matches new-object descriptors to demonstrations and jointly recovers the frame.This avoids the separate feature-matching and frame-recovery stages used by prior approaches.
- Equivariance: SE(3)-equivariant pose descriptor fields guarantee generalization across all 6-DoF configurations, including unseen translations and rotations.The design targets local-frame estimation for test-time objects in arbitrary configurations.
- Results: Above 85% overall success was achieved on three pick-and-place tasks using 10 expert demonstrations, outperforming 2D non-equivariant baselines.The result covers unseen object instances in out-of-distribution configurations.
II. METHOD
The method builds dense, category-level correspondence from 3D point clouds and uses demonstrations to define task-relevant local frames. Pose descriptor matching then transfers grasp and placement interactions to new objects.
- Representation: The representation models dense correspondence across object instances at the level of points and local coordinate frames.It provides an intuitive way to specify and compute task-relevant frames from demonstrations and point clouds.
- Point Descriptor Fields: A neural network learns point descriptors task-agnostically through 3D reconstruction, including point-wise correspondence across category-level shapes.The point descriptor field maps a 3D coordinate and point cloud to a spatial descriptor.
- Demonstration Encoding: Contact interactions with rigid bodies such as grippers, racks, and shelves provide query points near important geometric features directly from demonstrations.These descriptors define task-relevant local reference frames for grasping and placing.
A. Neural Point Descriptor Fields
Neural Point Descriptor Fields represent object point clouds as continuous functions that map 3D coordinates to category-level spatial descriptors. They use reconstruction-derived multiscale features and SE(3)-equivariant design to support correspondence across object instances and configurations.
- Neural Point Descriptor Fields: NDFs represent an object point cloud P as a continuous function f(x|P) that maps any 3D coordinate to a spatial descriptor.The coordinate need not lie on the object and may be occluded.
- Pose Descriptor Fields: For pose descriptors, NDFs transform a query point cloud, extract point-level descriptors, and concatenate them into a final pose descriptor Z.An energy-field minimizer can transfer a reference point to the corresponding feature on a test-time object.
- Neural Point Descriptor Fields: The descriptor field encodes each coordinate’s spatial relationship to salient geometric features such as a mug’s handle and rim.This category-level information is intended to remain consistent across object shapes.
- Neural Point Descriptor Fields: Category-level 3D reconstruction trains Φ as a hierarchical coarse-to-fine feature extractor whose activations capture increasingly fine surface detail.The encoder’s latent bottleneck determines where the object’s decision boundary lies and parameterizes salient category features.
- Neural Point Descriptor Fields: The point descriptor field f concatenates activations from all layers of Φ to encourage representations spanning multiple feature scales.The ith layer activation is denoted Φi, and the concatenation covers L layers.
- Equivariance w.r.t. SE(3): SE(3) equivariance makes descriptors transform consistently under joint object-coordinate transformations, enabling generalization to arbitrary unseen object poses.Translation equivariance uses point-cloud mean-centering, while rotation equivariance uses an SO(3)-equivariant architecture.
B. Neural Pose Descriptor Fields
Neural Pose Descriptor Fields extend point descriptors to encode SE(3) poses of external bodies relative to task-relevant object features. Pose transfer is performed by optimizing query-point transformations so their descriptors match those from demonstrations.
- Task-specific alignment: NDF pose regression transfers both position and orientation by recovering an equivalent pose for a gripper or other rigid body on a new object instance.This is necessary for tasks such as rim grasping and hanging a mug by its handle, where orientation is essential.
- Pose representation: Pose descriptor fields encode an external rigid body pose through descriptors concatenated over transformed query points.The query points are initialized canonically, transformed by an SE(3) pose, and evaluated with the point descriptor field.
- Pose representation: NDFs map an object point cloud and an SE(3) transformation to a category-level pose descriptor that inherits SE(3)-equivariance.This supports generalization across all 6-DoF configurations of the object point cloud.
- Task-specific alignment: Query-point locations determine which object features the transferred pose aligns to, making pose transfer task-specific.Points near a mug handle emphasize the handle, while other placements can align different geometric features.
- Pose transfer: Pose transfer optimizes the SE(3) transformation of query points to minimize descriptor distance from the demonstration pose.Translation and axis-angle rotation are optimized from a random initialization using descriptor differences back-propagated through the field.
C. Few-shot imitation learning with NDFs
NDFs acquire category-level pick-and-place skills by averaging pose descriptors across a small set of demonstrations and optimizing test-time poses to match them. The resulting poses are executed with standard inverse kinematics and motion planning.
- Demonstration encoding: Each demonstration contains an object point cloud, a pre-grasp end-effector pose, and a relative pose connecting grasp and placement.The demonstrations may use potentially partial object point clouds.
- Demonstration encoding: The method encodes gripper and placement poses with separate query-point sets and corresponding descriptor vectors.The descriptors are conditioned on each demonstration object point cloud.
- Few-shot skill acquisition: Pose descriptors are averaged across K demonstrations to produce single pick and placement targets for the object category.Averaging yields the reference descriptors used during test-time pose recovery.
- Few-shot skill acquisition: For a new object, test-time poses are recovered by minimizing distance to the averaged pick and placement descriptors.The predicted poses are then used to execute the transferred pick-and-place behavior.
- Execution: The final predicted pick-and-place task is executed with off-the-shelf inverse kinematics and motion planning algorithms.NDF optimization supplies the poses; conventional robot motion software performs execution.
III. EXPERIMENTS: DESIGN AND SETUP
The experiments test few-shot manipulation across unseen object instances and poses, compare NDFs with DON, study query-point effects, and evaluate transfer to real hardware. Simulation covers three household-object tasks and measures grasping, placing, and overall success.
- Evaluation design: The evaluation asks whether NDFs generalize manipulation to unseen objects and poses, how parameterization affects performance, and whether they transfer to a real robot.The setup uses a Franka Panda arm and depth cameras.
- Evaluation design: Fig. 6 tests how query-point placement near mug rims, handles, or bottoms changes aligned features and grasp outcomes.Rim-centered points can preserve a successful grasp on taller mugs, whereas bottom-centered points can bias the pose into collision.
- Evaluation design: Fig. 7 illustrates recovering an unseen-object gripper pose by minimizing the difference between demonstration and test pose descriptors.The gripper query point cloud defines the pose descriptor used for optimization.
- Tasks and assumptions: The simulated and real tasks include mug hanging, bowl placement, and bottle placement, under the assumption of segmented object clouds and a fixed environment.The mug is grasped by the rim and hung by its handle; bowls and bottles are placed upright on a shelf.
- Expected comparison: In simulation, NDFs match DON on upright grasp success, exceed DON on upright overall success, and retain higher success than DON for arbitrary object poses.The caption attributes the arbitrary-pose advantage to NDF SE(3) equivariance.
- Baselines and metrics: Success rates measure grasping, placing, and overall success, with overall success requiring both grasp and placement success.Training uses 100,000 randomly posed objects across mug, bowl, and bottle categories.
IV. EXPERIMENTS: RESULTS
The results examine NDF parameterization, robustness to arbitrary object poses, and real-robot transfer. Qualitative examples show that NDFs infer successful grasps on arbitrarily posed mugs where DON fails to detect the correct grasp keypoint.
- Results: Experiments compare NDF performance across object classes and pose configurations while ablating descriptor parameterization and demonstration count.The study includes comparisons with Dense Object Nets and evaluates how many demonstrations are needed.
- Qualitative results: For arbitrarily posed mugs, NDF predicts successful grasps while DON fails to detect the correct grasp keypoint.Both methods predict successful grasps when mugs remain upright.
- Real-robot transfer: The full model is also evaluated on a real robot to validate generalization to out-of-distribution object configurations.This complements the simulation studies of unseen object instances and poses.
A. Simulation Experiments
NDFs outperform DON in upright-pose transfer across mugs, bowls, and bottles, while descriptor matching provides greater robustness to shape and size variation. The harder arbitrary-SE(3) setting evaluates transfer from upright demonstrations to objects in arbitrary poses.
- Upright Pose: NDFs dramatically outperform DON on placing and perform significantly better on grasping across mugs, bowls, and bottles in upright poses.DON failures commonly involve imprecise keypoints, failed registration, or relative keypoint arrangements that differ across object shapes and sizes.
- Upright Pose: NDFs match descriptors in a learned, highly over-parameterized latent space, making placement-pose solving significantly more robust than keypoint-based transfer.The reported robustness addresses variation in object size and shape that can make semantically correct keypoints insufficient for successful placement.
- Arbitrary Pose: The arbitrary-pose evaluation tests whether upright demonstrations transfer to objects subsequently presented in arbitrary SE(3) poses.This is described as a harder setting than transfer among different upright poses.
B. Analysis
The analysis studies descriptor parameterization, query-point-cloud scale, and demonstration count, finding that full occupancy-network activations and task-aligned sampling are important design choices. It also examines how performance changes with fewer demonstrations.
- Neural Descriptors: Full NDF descriptors formed by concatenating all occupancy-network layer activations achieve the best performance by a large margin.The comparison includes a randomly initialized network and descriptors using only the first or last layer of a trained network.
- Query Point Cloud Scaling: Sampling query points within the bounding box of the interacting rigid body is a robust heuristic for representing grasping and placing pose descriptors.Scaling the query point cloud up or down reduces performance.
- Number of Demonstrations: The analysis evaluates NDF performance as the number of task demonstrations changes on the upright mug task.This experiment is part of the study of NDF dependence on demonstrations and descriptor construction.
C. Real World Execution
On a real robot, NDFs transfer pick-and-place tasks from ten upright demonstrations per category to novel mugs, bowls, and bottles in varied configurations.
- C. Real World Execution: Ten pick-and-place demonstrations on upright mugs, bowls, and bottles were transferred to novel real-object instances in varied, often challenging configurations.The executions included novel object instances and configurations differing from the demonstrations.
- C. Real World Execution: The real-robot validation demonstrates manipulation of novel object instances in novel poses.The paper directs readers to Fig. 9 and supplementary videos for visualizations and sample executions.
- C. Real World Execution: NDFs were evaluated on mugs, bowls, and bottles using demonstrations recorded in upright poses.The same pick-and-place task was then executed on novel instances across different configurations.
V. RELATED WORK
Prior manipulation approaches include pose estimation, primitive matching, non-rigid registration, direct policy learning, and category-level keypoints. NDFs instead transfer manipulation poses with self-supervised, task-agnostic descriptors and report stronger performance than 2D-correspondence baselines.
- V. RELATED WORK: Pose estimation supports known-object manipulation but does not enable category-level manipulation, while primitive matching and non-rigid registration can fail under substantial shape or scene differences.Directly learned pick-and-place policies require large amounts of demonstration data.
- V. RELATED WORK: Category-level keypoint methods may require human annotation or careful self-supervised correspondence selection, with manipulation outcomes sensitive to keypoint choice and accuracy.The passage contrasts directly predicted keypoints with self-supervised category-level correspondences.
- V. RELATED WORK: NDFs transfer observed manipulation poses across an object category using task-agnostic self-supervised pre-training without human-labeled keypoints.The representation is fully equivariant to SE(3) transformations.
- V. RELATED WORK: NDFs significantly outperform baselines based on correspondence predicted in 2D when imitating full pick-and-place tasks for unseen object configurations.The contribution statement describes this result together with transfer from a small handful of demonstrations.
B. Neural Fields and Neural Scene Representations
Neural Descriptor Fields build on neural implicit representations to encode continuous 3D object descriptors, exploiting their resolution, functional structure, and compatibility with geometric symmetries. The paper applies these capabilities to manipulation representations that generalize across object instances and SE(3) configurations, while identifying boundaries for non-rigid objects, full trajectories, and moving targets.
- Neural implicit representations: Neural implicit representations parameterize continuous fields for 3D geometry, appearance, and tactile properties.Their continuous nature provides effectively infinite-resolution scene-surface representations.
- Neural implicit representations: Functional neural fields support principled incorporation of symmetries such as SO(3) equivariance and latent spaces encoding class information and 3D correspondence.They have also been used to estimate unknown camera poses in 3D reconstruction.
- Limitations and future work: The approach remains untested for non-rigid objects, and transferring full trajectories or handling non-static placement targets requires future extensions.The authors suggest non-rigid scene methods, trajectory optimization, and object-centric target representations as possible directions.
- Neural Descriptor Fields: Neural Descriptor Fields extend these ideas into object representations that support few-shot manipulation imitation after task-agnostic 3D geometry reconstruction, without further imitation-time training.The representation is designed to generalize across object instances and SE(3) configurations.