Source-linked AI summary
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
Mohit Shridhar, Lucas Manuelli, Dieter Fox
TL;DR
Robotic manipulation has limited and expensive data, raising whether Transformers can be made data-efficient for 6-DoF control. PERACT addresses this with language-conditioned behavior cloning over RGB-D voxels and discretized voxel actions, and it significantly outperforms image-to-action and 3D ConvNet baselines across simulated and real-world tasks.
Problem
Robotic manipulation has limited and expensive data, creating a question about whether Transformers can benefit 6-DoF manipulation with an appropriate problem formulation.
Method
PERACT uses a Perceiver Transformer to encode language goals and RGB-D voxel observations, then predicts the next best discretized translation, rotation, and gripper action.
Results
PERACT significantly outperforms image-to-action agents by 34× and 3D ConvNet baselines by 2.8× across 18 RLBench tasks with 249 variations.
Takeaways & Limitations
Voxel action detection makes a substantial difference in data efficiency and robustness for simulated and real-world 6-DoF manipulation tasks.
Takeaways & Limitations
Extending PERACT to dexterous continuous control remains challenging because it relies on a sampling-based motion planner and is not easily extended to N-DoF actuators such as multi-fingered hands.
Abstract
from arXiv · showhide
Transformers have revolutionized vision and natural language processing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with PerAct, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. PerAct encodes language goals and RGB-D voxel observations with a Perceiver Transformer, and outputs discretized actions by ``detecting the next best voxel action''. Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides a strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PerAct significantly outperforms unstructured image-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.
1 Introduction
The paper asks whether Transformers can support data-efficient 6-DoF manipulation despite limited, costly robotic data, and proposes PERACT as a language-conditioned voxel-based behavior-cloning agent. PERACT uses a Perceiver Transformer and achieves strong multi-task results across simulated and real-world manipulation settings.
- Robotic manipulation presents a data-efficiency challenge because demonstrations are limited and expensive, motivating a problem formulation that makes Transformers practical for 6-DoF actions.
- PERACT is a language-conditioned behavior-cloning agent that encodes RGB-D voxel observations and predicts discretized translations, rotations, and gripper actions.Its actions are executed with a motion-planner in an observe-act loop.
- The approach is also demonstrated on 7 real-world tasks with a Franka Panda using a multi-task agent trained from 53 demonstrations.
- Perceiver processes high-dimensional voxel inputs with a small set of latent vectors, while voxelization supports multi-view fusion, action-centric representations, and 6-DoF data augmentation.
- PERACT trains a single multi-task agent on 18 RLBench tasks with 249 variations, spanning diverse prehensile and non-prehensile behaviors.The tasks include pose and semantic variations in object placement, color, shape, size, and category.
- PERACT significantly outperforms image-to-action agents by 34× and 3D ConvNet baselines by 2.8× in the reported RLBench experiments.
2 Related Work
Prior manipulation systems often rely on object representations, 2D image-to-action prediction, or language-guided perception pipelines. Related work motivates PERACT’s combination of action-centric representations, 3D voxelization, and end-to-end language-conditioned control.
- Traditional robot-perception methods use instance segmentations, object classes, and poses, but these representations are difficult for deformable and granular items such as cloth and beans.
- Action-centric approaches avoid objectness assumptions but have generally been limited to top-down 2D settings with simple pick-and-place primitives.
- End-to-end image-to-action methods make few object and task assumptions, yet direct RGB training for 6-DoF manipulation is often inefficient.Such methods may require several demonstrations or episodes to learn basic rearrangement skills.
- Transformers have been applied across robotics domains including teleoperation, locomotion, planning, imitation learning, rearrangement, and grasping.
- Language-grounding methods commonly use disentangled perception and action pipelines, while newer approaches condition behavior-cloning agents directly on language instructions.
3 PERCEIVER-ACTOR
PERACT formulates language-conditioned 6-DoF manipulation as next-keyframe action detection over voxelized RGB-D observations. A Perceiver Transformer encodes language and 3D voxel patches, predicts discretized actions, and executes them iteratively with a motion planner.
- PERACT predicts the next best discretized 6-DoF action from a voxelized observation and language goal at each timestep.Its outputs include translation, rotation, gripper state, and collision-avoidance decisions.
- Expert demonstrations are paired with English goals and contain RGB-D observations, continuous actions, and motion-planner-assisted intermediate poses.Simulated experiments use four cameras, while real-world experiments use one camera.
- 3.2 Keyframes and Voxelization: Keyframe extraction converts continuous demonstrations into bottleneck end-effector poses, defined by near-zero joint velocities and unchanged gripper state.Training then becomes a next-best-keyframe prediction task rather than direct continuous-action prediction.
- 3.2 Keyframes and Voxelization: RGB-D observations are fused into a 100^3 voxel grid, and keyframe actions are discretized so behavior cloning becomes voxel-based action classification.The voxel space represents both observations and actions, providing a structured 3D formulation.
- 3.3 PERACT Agent: Language and 3D voxel patches are encoded together with a Perceiver Transformer that uses a small latent set for the extremely long input sequence.The decoder upsamples patch encodings into per-voxel features used for discretized action prediction.
- 3.4 Training Details: Supervised training samples voxel-observation, language-goal, and keyframe tuples, applies cross-entropy to discrete action outputs, and augments observations and actions with pose perturbations.Multi-task batches sample tasks uniformly before selecting an input-action tuple.
4 Results
Experiments evaluate PERACT against image-based and 3D convolutional baselines across simulated and real-world manipulation tasks, while probing component, receptive-field, and sensitivity effects. PERACT generally performs strongly with few demonstrations, but performance remains difficult on high-precision or highly varied tasks.
- Experimental Questions: The evaluation tests PERACT against image-to-action and 3D ConvNet baselines, examines global versus local receptive fields, and studies factors affecting performance.The experiments address simulated reproducibility, multi-task benchmarking, ablations, sensitivity, and real-world training.
- Simulation Results: PERACT outperforms C2FARM-BC with average improvements of 1.33× using 10 demonstrations and 2.83× using 100 demonstrations.Table 1 evaluates agents on 18 tasks, using 25 episodes per task and binary success scoring.
- Simulation Results: PERACT outperforms C2FARM-BC in 25/36 evaluations, while Image-BC achieves near-zero performance on most tasks with insufficient demonstrations.The voxel formulation supports multi-view integration, 6-DoF action representations, and 3D data augmentation, whereas C2FARM-BC has a more limited receptive field.
- Simulation Results: 10 demonstrations are sufficient for PERACT to exceed 65% success on limited-variation tasks such as open drawer, whereas highly varied tasks require substantially more data.Some high-precision tasks remain near zero for all agents, though single-task training slightly alleviates this issue.
- Ablations: Ablations show that language conditioning is necessary for task identification, the Perceiver supports global receptive fields, and keyframe extraction is essential for supervised training.The skip connection only helps training speed slightly; random or fixed-interval keyframes produce zero performance.
- Global vs. Local Receptive Fields: PERACT is the only agent exceeding 70% success on open drawer, while all C2FARM-BC variants perform near chance at approximately 33%.The experiment uses 100 demonstrations and tests whether a global receptive field helps distinguish visually identical drawer handles.
5 Limitations and Conclusion
PERACT is presented as a Transformer-based multi-task agent whose voxel-action formulation improves data efficiency and robustness across simulated and real-world manipulation. Its main scope boundary is dexterous continuous control.
- Conclusion: PERACT is a Transformer-based multi-task agent for 6-DoF manipulation.
- Conclusion: Experiments in simulated and real-world tasks indicate that detecting voxel actions substantially improves data efficiency and robustness.
- Limitations: Extending PERACT to dexterous continuous control remains challenging because it relies on a sampling-based motion planner for discretized actions and does not readily extend to N-DoF actuators.
A Task Details
The task suite evaluates language-conditioned manipulation across diverse RLBench variations, with explicit task objectives, objects, and success criteria. Implementation details describe PERACT’s voxel-processing architecture and Perceiver-based encoding.
- Task Setup: The evaluation selects 18 RLBench tasks with at least two variations to test multi-task language grounding rather than single-variation policies.
- Task Variations: Task variations randomize colors, sizes, shapes, counts, placements, and object categories to diversify language-conditioned evaluation.
- Task Details: The suite includes object-placement and manipulation tasks such as opening drawers, sliding blocks, sweeping dirt, handling meat, and turning faucet handles.
- Success Metrics: Success criteria specify the required final object configuration, including containment, placement location, joint displacement, or button activation.
- Task Details: Other tasks require placing objects in drawers, jars, safes, racks, cupboards, sorters, or on specified platforms and spokes.
- Implementation: Perceiver latents are encoded with 6 self-attention layers, then cross-attended with the input and upsampled into a 100×100×100×64 voxel feature grid.
E Data Augmentation
PERACT trains by predicting the next best keyframe action from voxelized demonstrations and augments observations and actions with bounded spatial transformations.
- SE(3) Augmentation: Training applies random translations and yaw rotations jointly to voxelized observations and their corresponding keyframe actions.
- SE(3) Augmentation: Translation perturbations reach ±0.125m per axis, while yaw perturbations are limited to ±45° to remain kinematically reasonable.
- Keyframe Prediction: Each demonstration datapoint is framed as predicting the next best keyframe action.
G Sensitivity Analysis
The sensitivity analysis varies rotation augmentation, Perceiver latent count, and voxelization resolution to identify factors affecting PERACT’s multi-task performance. The reported effects depend on task rotation and precision requirements.
- Sensitivity Analysis: Table 4 evaluates rotation augmentation, the number of Perceiver latents, and voxelization resolution using agents trained with 100 demonstrations per task.
- Rotation Augmentation: 45° yaw perturbations improve tasks with many rotation variations but worsen tasks with constrained rotations such as place wine.
- Voxelization Resolution: Tasks requiring finer spatial precision need higher-resolution voxelization, while larger patch sizes reduce memory usage but may harm sub-patch precision.
H High-Precision Tasks
PERACT achieves zero performance on three high-precision tasks, while separate single-task agents achieve non-zero performance, implicating multi-task optimization as a factor.
- PERACT achieves zero performance on place cups, stack cups, and insert peg.
- Separate single-task agents achieve non-zero performance on those high-precision tasks.The comparison suggests that improved multi-task optimization could improve performance on these tasks.
- The paper identifies better multi-task optimization methods as a possible direction for improving difficult high-precision tasks.
J Additional Q-Prediction Examples
Additional Q-prediction examples illustrate PERACT’s action-centric representation, which focuses on task-relevant actions rather than requiring explicit object definitions.
- Figure 10 presents additional Q-prediction examples from trained PERACT agents.
- Action-centric agents focus on perceptual representations of actions instead of assuming explicit object representations.
- This approach reduces the need for practitioners to define objects, which can be difficult and task- or embodiment-specific.
K Things that did not work
Several design choices failed or created practical problems, including multi-camera interference, slower training, worse optimization, and difficulties with aggressive rotation augmentation.
- Real-world multi-camera setup: Multiple Kinect-2 cameras produced noisy depth frames with many holes because of unresolved Time-of-Flight interference.
- Fourier features for positional embeddings: Fourier features produced substantially worse performance than learned positional embeddings.
- Pre-trained vision features: Using CLIP vision features was extremely slow because ResNet50 and decoder layers had to process four independent RGB frames.
- Upsampling at multiple self-attention layers: Upsampling features at multiple self-attention layers did not work at all.
- Extreme rotation augmentation: Pitch-and-roll augmentation still learned policies but substantially increased training time, with latent-size suitability remaining unclear.
- Using Adam instead of LAMB: Adam led to worse performance than LAMB in both simulated and real-world experiments.
L Limitations and Risks
PERACT’s limitations concern execution, dynamics, dexterity, generalization, language scope, observability, data balance, task weighting, and deployment safety.
- Sampling-Based Motion Planner: Sampling-based planning makes execution sensitive to randomized paths, which can matter for tasks requiring smooth motion such as pouring.
- Dynamic Manipulation: Discrete-time actions are not easily applicable to dynamic tasks requiring real-time closed-loop maneuvering.
- Dexterous Manipulation: Discretized actions are non-trivial for multi-fingered hands, especially under-actuated systems where feasibility and robustness remain unclear.
- Generlization to Novel Instances and Objects: Randomized drawer textures and colors confuse the agent when training exposes it to only one drawer appearance.
- Scope of Language Grounding: PERACT’s verb-noun grounding is tied closely to demonstrated tasks rather than general meanings of verbs such as “cleaning.”
- Predicting Task Completion: Current evaluations use an oracle to indicate task completion rather than a learned success classifier.
- History and Partial Observability: Current-observation-only prediction makes history-dependent and partially observable tasks infeasible without additional completion or memory mechanisms.
- Multi-Task Optimization: Uniform task sampling can hurt dissimilar tasks when optimization favors tasks sharing common elements.