Source-linked AI summary
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, Katerina Fragkiadaki
TL;DR
High-resolution 3D action maps are costly, limiting spatially grounded manipulation policies. Act3D uses an adaptive coarse-to-fine 3D feature field built from pretrained 2D features and depth, achieving large RLBench gains over prior 2D and 3D policies.
Problem
High-resolution 3D action maps are computationally expensive, while 2D policies can require many demonstrations and generalize poorly across scene and viewpoint changes.
Method
Act3D lifts pretrained 2D visual features into 3D and recurrently samples, featurizes, and scores workspace points from coarse to fine using relative spatial attention.
Results
Act3D achieves a 10% absolute improvement over prior SOTA in single-task RLBench and a 22% absolute improvement in multi-task RLBench.
Takeaways & Limitations
Act3D provides continuous-resolution 3D action maps while preserving spatial locality and reports strong results across simulated and real-world manipulation tasks.
Takeaways & Limitations
Act3D is limited by straight-segment motion planning for articulated-object manipulation and does not decompose tasks into subtasks.
Abstract
from arXiv · showhide
3D perceptual representations are well suited for robot manipulation as they easily encode occlusions and simplify spatial reasoning. Many manipulation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that are computationally expensive to process. As a result, most manipulation policies operate directly in 2D, foregoing 3D inductive biases. In this paper, we introduce Act3D, a manipulation policy transformer that represents the robot's workspace using a 3D feature field with adaptive resolutions dependent on the task at hand. The model lifts 2D pre-trained features to 3D using sensed depth, and attends to them to compute features for sampled 3D points. It samples 3D point grids in a coarse to fine manner, featurizes them using relative-position attention, and selects where to focus the next round of point sampling. In this way, it efficiently computes 3D action maps of high spatial resolution. Act3D sets a new state-of-the-art in RL-Bench, an established manipulation benchmark, where it achieves 10% absolute improvement over the previous SOTA 2D multi-view policy on 74 RLBench tasks and 22% absolute improvement with 3x less compute over the previous SOTA 3D policy. We quantify the importance of relative spatial attention, large-scale vision-language pre-trained 2D backbones, and weight tying across coarse-to-fine attentions in ablative experiments. Code and videos are available on our project website: https://act3d.github.io/.
1 Introduction
Act3D addresses the computational cost of high-resolution 3D action maps for spatially equivariant 6-DoF manipulation with adaptive coarse-to-fine 3D computation. It achieves state-of-the-art RLBench results and validates several architectural choices.
- Motivation: 2D policies often require many trajectories for minor scene changes and generalize poorly across camera viewpoints and environments.
- Motivation: Spatial equivariance maps scene translations and rotations to corresponding end-effector transformations, requiring 2D or 3D action maps rather than holistic-feature regression.
- Challenge: High-resolution 3D action maps improve fine-grained manipulation but are computationally demanding because the workspace includes an extra spatial dimension and unconstrained free space.
- Approach: Act3D learns continuous-resolution 3D action maps by lifting pretrained 2D features with depth and recurrently sampling and featurizing workspace points from coarse to fine.
- Results: 10% absolute improvement over prior SOTA in single-task RLBench and 22% absolute improvement over prior SOTA in multi-task RLBench establish new benchmark results.The comparisons cover 74 single-task activities and 18 multi-task tasks with 249 variations.
- Analysis: Ablations quantify the contributions of high-resolution features, tied attention weights, pretrained 2D features, and relative-position attention.
2 Related Work
Prior work spans end-to-end 2D image-to-action policies, pretrained visual backbones, and relative-attention architectures. Act3D combines these directions with 3D spatial reasoning for manipulation.
- Learning robot manipulation from demonstrations: End-to-end policies such as RT-1, GATO, BC-Z, and InstructRL predict 6-DoF poses from 2D video and language but require many demonstrations for spatial reasoning and generalization.
- Feature pre-training for robot manipulation: Pretrained 2D vision-language backbones improve demonstration efficiency and can support generalization to new instructions, objects, and scenes.
- Relative attention layers: Relative attention is especially important in 3D because the coordinate system is arbitrary, and prior 3D Transformers have used it for detection and point labelling.
3 3D Feature Field Transformers for Multi-Task Robot Manipulation
Act3D predicts keypose actions from RGB-D observations, language, and proprioception using a continuous 3D feature field. Its recurrent coarse-to-fine sampling and relative attention allocate computation across workspace regions.
- Policy formulation: Act3D predicts the next 6-DoF keypose from RGB-D views, a language instruction, and the robot’s current end-effector pose.Keyposes capture state changes or near-zero-velocity transition points, and a motion planner reaches each predicted keypose.
- Policy formulation: Training actions include end-effector position, quaternion orientation, binary gripper state, and a collision-avoidance flag.
- Visual and language encoder: The visual encoder extracts multiscale 2D tokens and lifts them into a 3D scene feature cloud using pixel depth and camera intrinsics.
- Iterative 3D point sampling and featurization: Ghost points are sampled across the workspace, then locally refined, and iteratively scored through relative cross-attention to the lifted scene cloud.
- Decoding actions: The decoder selects the highest-scoring ghost point as the end-effector position and regresses orientation, opening, and collision avoidance from the final query.
- Implementation details: Training uses 1,000 ghost points across three stages, while inference can use 10,000 points to trade additional compute for precision and task performance.
4 Experiments
Act3D is evaluated on RLBench in single-task and multi-task simulation settings, with additional viewpoint, ablation, and real-world experiments. It achieves strong benchmark performance, improves robustness to camera changes, and reveals the importance of several architectural choices.
- Single-task results: Act3D reaches 83% success on 74 single-task RLBench tasks, a 10% absolute improvement over InstructRL.The tasks span 9 categories; Act3D also remains competitive with only 10 demonstrations per task.
- Multi-task results: Act3D reaches 65% success on 18 multi-task RLBench tasks with 249 variations, a 22% absolute improvement over PerAct.This result uses less than a third of PerAct’s training computation budget: 5 days versus 16 days on 8 Nvidia V100 GPUs.
- Camera-viewpoint generalization: Act3D achieves 74.2% success under novel camera viewpoints, while HiveFormer reaches 20.4%.The corresponding relative drops are 24% for Act3D and 77% for HiveFormer.
- Ablations: Removing one coarse-to-fine sampling stage reduces performance by 4.5%, while removing weight tying or relative positional embeddings causes larger drops.The reported drops are 17.5% without weight tying and 42.7% without relative 3D positional embeddings.
- Ablations: A CLIP-pre-trained ResNet50 improves multi-task success rate by 8.7% over an ImageNet-pre-trained ResNet50.The backbone comparison uses all 18 multi-task tasks.
- Real-world evaluation: In a real-world study, Act3D was trained on 8 Franka Panda tasks using 10 to 15 kinesthetic demonstrations per task and evaluated for 10 episodes per task.The tasks include liquid, articulated, and deformable objects, using a single Azure Kinect RGB-D sensor.
- Limitations: Act3D is limited by its straight-segment motion planner for articulated-object manipulation and lacks task decomposition into reusable subtasks.The paper identifies hierarchical language subgoals as a direction for future work.
5 Conclusion
Act3D is a language-conditioned policy transformer for continuous-resolution 3D action maps in multi-task robot manipulation. It achieves state-of-the-art RLBench performance, addresses diverse real-world tasks from one RGB-D view and few demonstrations, and is supported by ablations.
- Act3D predicts continuous-resolution 3D action maps for multi-task robot manipulation.
- The method represents scenes with coarse-to-fine 3D point sampling and attention-based featurization.
- Act3D sets a new state-of-the-art in RLBench and solves diverse real-world tasks from a single RGB-D camera view with few demonstrations.
- Ablations quantify the contributions of relative 3D attention, 2D feature pre-training, and weight tying across coarse-to-fine iterations.
7 Appendix
The appendix details Act3D’s simulation and real-world setups, task organization, relative 3D attention, adaptive point sampling, high-precision experiments, and ablations.
- 7.2 RLBench Simulation Setup: The real-world setup uses a Franka Panda with one Azure Kinect RGB-D sensor, while simulated evaluations match prior work’s camera configurations.
- 7.4 Further Architecture Details: Act3D independently featurizes ghost points and queries through relative 3D cross-attention to scene features, language, and proprioception.
- 7.4 Further Architecture Details: Three sampling stages move from the roughly 1-meter workspace to 16-centimeter and 4-centimeter regions, using coarse global and fine local feature clouds.
- 7.5 High Precision Experiments: Act3D improves over PerAct on high-precision tasks and benefits from higher-resolution RGB-D images at extra compute cost.
- 7.6 Further ablations: Random crops increase success rate by 6.5%, whereas yaw rotation perturbations decrease it by 11.9%.