Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation

Motivation

• 2D policies often require many trajectories for minor scene changes and generalize poorly across camera viewpoints and environments.Source: §1 · p. 1
• Spatial equivariance maps scene translations and rotations to corresponding end-effector transformations, requiring 2D or 3D action maps rather than holistic-feature regression.Source: §1 · p. 1

Challenge

• High-resolution 3D action maps improve fine-grained manipulation but are computationally demanding because the workspace includes an extra spatial dimension and unconstrained free space.Source: §1 · p. 2

Approach

• Act3D learns continuous-resolution 3D action maps by lifting pretrained 2D features with depth and recurrently sampling and featurizing workspace points from coarse to fine.Source: §1 · p. 2, Figure 1 · p. 2

Results

• 10% absolute improvement over prior SOTA in single-task RLBench and 22% absolute improvement over prior SOTA in multi-task RLBench establish new benchmark results.The comparisons cover 74 single-task activities and 18 multi-task tasks with 249 variations.Source: §1 · p. 2

Analysis

• Ablations quantify the contributions of high-resolution features, tied attention weights, pretrained 2D features, and relative-position attention.Source: §1 · p. 3

Learning robot manipulation from demonstrations

• End-to-end policies such as RT-1, GATO, BC-Z, and InstructRL predict 6-DoF poses from 2D video and language but require many demonstrations for spatial reasoning and generalization.Source: §2 · p. 3

Feature pre-training for robot manipulation

• Pretrained 2D vision-language backbones improve demonstration efficiency and can support generalization to new instructions, objects, and scenes.Source: §2 · p. 3

Relative attention layers

• Relative attention is especially important in 3D because the coordinate system is arbitrary, and prior 3D Transformers have used it for detection and point labelling.Source: §2 · p. 3

Policy formulation

• Act3D predicts the next 6-DoF keypose from RGB-D views, a language instruction, and the robot’s current end-effector pose.Keyposes capture state changes or near-zero-velocity transition points, and a motion planner reaches each predicted keypose.Source: §3 · pp. 3–4
• Training actions include end-effector position, quaternion orientation, binary gripper state, and a collision-avoidance flag.Source: §3 · p. 4

Visual and language encoder

• The visual encoder extracts multiscale 2D tokens and lifts them into a 3D scene feature cloud using pixel depth and camera intrinsics.Source: §3 · p. 4

Iterative 3D point sampling and featurization

• Ghost points are sampled across the workspace, then locally refined, and iteratively scored through relative cross-attention to the lifted scene cloud.Source: §3 · p. 4

Decoding actions

• The decoder selects the highest-scoring ghost point as the end-effector position and regresses orientation, opening, and collision avoidance from the final query.Source: §3 · p. 4

Implementation details

• Training uses 1,000 ghost points across three stages, while inference can use 10,000 points to trade additional compute for precision and task performance.Source: §3 · p. 5

Single-task results

• Act3D reaches 83% success on 74 single-task RLBench tasks, a 10% absolute improvement over InstructRL.The tasks span 9 categories; Act3D also remains competitive with only 10 demonstrations per task.Source: Figure 3 · p. 6, §4.1 · p. 7

Multi-task results

• Act3D reaches 65% success on 18 multi-task RLBench tasks with 249 variations, a 22% absolute improvement over PerAct.This result uses less than a third of PerAct’s training computation budget: 5 days versus 16 days on 8 Nvidia V100 GPUs.Source: Figure 4 · p. 6, §4.1 · p. 7

Camera-viewpoint generalization

• Act3D achieves 74.2% success under novel camera viewpoints, while HiveFormer reaches 20.4%.The corresponding relative drops are 24% for Act3D and 77% for HiveFormer.Source: §4.2 · p. 7

Ablations

• Removing one coarse-to-fine sampling stage reduces performance by 4.5%, while removing weight tying or relative positional embeddings causes larger drops.The reported drops are 17.5% without weight tying and 42.7% without relative 3D positional embeddings.Source: §4.2 · p. 7
• A CLIP-pre-trained ResNet50 improves multi-task success rate by 8.7% over an ImageNet-pre-trained ResNet50.The backbone comparison uses all 18 multi-task tasks.Source: §4.2 · pp. 7–8

Real-world evaluation

• In a real-world study, Act3D was trained on 8 Franka Panda tasks using 10 to 15 kinesthetic demonstrations per task and evaluated for 10 episodes per task.The tasks include liquid, articulated, and deformable objects, using a single Azure Kinect RGB-D sensor.Source: §4.3 · p. 8

Limitations

• Act3D is limited by its straight-segment motion planner for articulated-object manipulation and lacks task decomposition into reusable subtasks.The paper identifies hierarchical language subgoals as a direction for future work.Source: §4.4 · p. 8

• Act3D predicts continuous-resolution 3D action maps for multi-task robot manipulation.Source: §5 · p. 8
• The method represents scenes with coarse-to-fine 3D point sampling and attention-based featurization.Source: §5 · p. 8
• Act3D sets a new state-of-the-art in RLBench and solves diverse real-world tasks from a single RGB-D camera view with few demonstrations.Source: §5 · p. 8
• Ablations quantify the contributions of relative 3D attention, 2D feature pre-training, and weight tying across coarse-to-fine iterations.Source: §5 · p. 8

7.2 RLBench Simulation Setup

• The real-world setup uses a Franka Panda with one Azure Kinect RGB-D sensor, while simulated evaluations match prior work’s camera configurations.Source: §7.1 · p. 12, §7.2 · p. 12

7.4 Further Architecture Details

• Act3D independently featurizes ghost points and queries through relative 3D cross-attention to scene features, language, and proprioception.Source: §7.4 · p. 14
• Three sampling stages move from the roughly 1-meter workspace to 16-centimeter and 4-centimeter regions, using coarse global and fine local feature clouds.Source: §7.4 · p. 14

7.5 High Precision Experiments

• Act3D improves over PerAct on high-precision tasks and benefits from higher-resolution RGB-D images at extra compute cost.Source: §7.5 · pp. 15–16

7.6 Further ablations

• Random crops increase success rate by 6.5%, whereas yaw rotation perturbations decrease it by 11.9%.Source: §7.6 · p. 16