Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

• Robotic manipulation presents a data-efficiency challenge because demonstrations are limited and expensive, motivating a problem formulation that makes Transformers practical for 6-DoF actions.Source: §1 · p. 1
• PERACT is a language-conditioned behavior-cloning agent that encodes RGB-D voxel observations and predicts discretized translations, rotations, and gripper actions.Its actions are executed with a motion-planner in an observe-act loop.Source: §1 · p. 1
• The approach is also demonstrated on 7 real-world tasks with a Franka Panda using a multi-task agent trained from 53 demonstrations.Source: Figure 1 · p. 2
• Perceiver processes high-dimensional voxel inputs with a small set of latent vectors, while voxelization supports multi-view fusion, action-centric representations, and 6-DoF data augmentation.Source: §1 · p. 2
• PERACT trains a single multi-task agent on 18 RLBench tasks with 249 variations, spanning diverse prehensile and non-prehensile behaviors.The tasks include pose and semantic variations in object placement, color, shape, size, and category.Source: §1 · p. 2
• PERACT significantly outperforms image-to-action agents by 34× and 3D ConvNet baselines by 2.8× in the reported RLBench experiments.Source: §1 · p. 2

• Traditional robot-perception methods use instance segmentations, object classes, and poses, but these representations are difficult for deformable and granular items such as cloth and beans.Source: §2 · p. 3
• Action-centric approaches avoid objectness assumptions but have generally been limited to top-down 2D settings with simple pick-and-place primitives.Source: §2 · p. 3
• End-to-end image-to-action methods make few object and task assumptions, yet direct RGB training for 6-DoF manipulation is often inefficient.Such methods may require several demonstrations or episodes to learn basic rearrangement skills.Source: §2 · p. 3
• Transformers have been applied across robotics domains including teleoperation, locomotion, planning, imitation learning, rearrangement, and grasping.Source: §2 · p. 3
• Language-grounding methods commonly use disentangled perception and action pipelines, while newer approaches condition behavior-cloning agents directly on language instructions.Source: §2 · p. 3

• PERACT predicts the next best discretized 6-DoF action from a voxelized observation and language goal at each timestep.Its outputs include translation, rotation, gripper state, and collision-avoidance decisions.Source: §3 · p. 3, §3.3 · pp. 4–5
• Expert demonstrations are paired with English goals and contain RGB-D observations, continuous actions, and motion-planner-assisted intermediate poses.Simulated experiments use four cameras, while real-world experiments use one camera.Source: §3.1 · p. 4

3.2 Keyframes and Voxelization

• Keyframe extraction converts continuous demonstrations into bottleneck end-effector poses, defined by near-zero joint velocities and unchanged gripper state.Training then becomes a next-best-keyframe prediction task rather than direct continuous-action prediction.Source: §3.2 · p. 4
• RGB-D observations are fused into a 100^3 voxel grid, and keyframe actions are discretized so behavior cloning becomes voxel-based action classification.The voxel space represents both observations and actions, providing a structured 3D formulation.Source: §3.2 · p. 4

3.3 PERACT Agent

• Language and 3D voxel patches are encoded together with a Perceiver Transformer that uses a small latent set for the extremely long input sequence.The decoder upsamples patch encodings into per-voxel features used for discretized action prediction.Source: §3.3 · pp. 4–5

3.4 Training Details

• Supervised training samples voxel-observation, language-goal, and keyframe tuples, applies cross-entropy to discrete action outputs, and augments observations and actions with pose perturbations.Multi-task batches sample tasks uniformly before selecting an input-action tuple.Source: §3.4 · p. 5

Experimental Questions

• The evaluation tests PERACT against image-to-action and 3D ConvNet baselines, examines global versus local receptive fields, and studies factors affecting performance.The experiments address simulated reproducibility, multi-task benchmarking, ablations, sensitivity, and real-world training.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 6

Simulation Results

• PERACT outperforms C2FARM-BC with average improvements of 1.33× using 10 demonstrations and 2.83× using 100 demonstrations.Table 1 evaluates agents on 18 tasks, using 25 episodes per task and binary success scoring.Source: §4.1 · p. 6, Table 1 · p. 7, §4.2 · p. 7
• PERACT outperforms C2FARM-BC in 25/36 evaluations, while Image-BC achieves near-zero performance on most tasks with insufficient demonstrations.The voxel formulation supports multi-view integration, 6-DoF action representations, and 3D data augmentation, whereas C2FARM-BC has a more limited receptive field.Source: §4.2 · pp. 6–7
• 10 demonstrations are sufficient for PERACT to exceed 65% success on limited-variation tasks such as open drawer, whereas highly varied tasks require substantially more data.Some high-precision tasks remain near zero for all agents, though single-task training slightly alleviates this issue.Source: §4.2 · p. 7

Ablations

• Ablations show that language conditioning is necessary for task identification, the Perceiver supports global receptive fields, and keyframe extraction is essential for supervised training.The skip connection only helps training speed slightly; random or fixed-interval keyframes produce zero performance.Source: §4.2 · p. 7

Global vs. Local Receptive Fields

• PERACT is the only agent exceeding 70% success on open drawer, while all C2FARM-BC variants perform near chance at approximately 33%.The experiment uses 100 demonstrations and tests whether a global receptive field helps distinguish visually identical drawer handles.Source: §4.3 · pp. 7–8, Figure 4 · p. 7

Conclusion

• PERACT is a Transformer-based multi-task agent for 6-DoF manipulation.Source: §5 · p. 8
• Experiments in simulated and real-world tasks indicate that detecting voxel actions substantially improves data efficiency and robustness.Source: §5 · p. 8

Limitations

• Extending PERACT to dexterous continuous control remains challenging because it relies on a sampling-based motion planner for discretized actions and does not readily extend to N-DoF actuators.Source: §5 · p. 8

Task Setup

• The evaluation selects 18 RLBench tasks with at least two variations to test multi-task language grounding rather than single-variation policies.Source: A Task Details · p. 16

Task Variations

• Task variations randomize colors, sizes, shapes, counts, placements, and object categories to diversify language-conditioned evaluation.Source: A Task Details · p. 16

Task Details

• The suite includes object-placement and manipulation tasks such as opening drawers, sliding blocks, sweeping dirt, handling meat, and turning faucet handles.Source: §A.1 · p. 16, §A.2 · p. 16, §A.3 · p. 17

Success Metrics

• Success criteria specify the required final object configuration, including containment, placement location, joint displacement, or button activation.Source: §A.1 · p. 16, §A.2 · p. 17, §A.5 · p. 17

Task Details

• Other tasks require placing objects in drawers, jars, safes, racks, cupboards, sorters, or on specified platforms and spokes.Source: §A.6 · p. 17, §A.7 · p. 17, §A.11 · p. 18

Implementation

• Perceiver latents are encoded with 6 self-attention layers, then cross-attended with the input and upsampled into a 100×100×100×64 voxel feature grid.Source: B PERACT Details · p. 20, Figure 6 · p. 20

SE(3) Augmentation

• Training applies random translations and yaw rotations jointly to voxelized observations and their corresponding keyframe actions.Source: E Data Augmentation · p. 23
• Translation perturbations reach ±0.125m per axis, while yaw perturbations are limited to ±45° to remain kinematically reasonable.Source: E Data Augmentation · p. 23

Keyframe Prediction

• Each demonstration datapoint is framed as predicting the next best keyframe action.Source: F Demo Augmentation · p. 23

Sensitivity Analysis

• Table 4 evaluates rotation augmentation, the number of Perceiver latents, and voxelization resolution using agents trained with 100 demonstrations per task.Source: G Sensitivity Analysis · p. 23, Table 4 · p. 23

Rotation Augmentation

• 45° yaw perturbations improve tasks with many rotation variations but worsen tasks with constrained rotations such as place wine.Source: G Sensitivity Analysis · p. 23

Voxelization Resolution

• Tasks requiring finer spatial precision need higher-resolution voxelization, while larger patch sizes reduce memory usage but may harm sub-patch precision.Source: G Sensitivity Analysis · p. 24

• PERACT achieves zero performance on place cups, stack cups, and insert peg.Source: H High-Precision Tasks · p. 24
• Separate single-task agents achieve non-zero performance on those high-precision tasks.The comparison suggests that improved multi-task optimization could improve performance on these tasks.Source: H High-Precision Tasks · p. 24
• The paper identifies better multi-task optimization methods as a possible direction for improving difficult high-precision tasks.Source: H High-Precision Tasks · p. 24

• Figure 10 presents additional Q-prediction examples from trained PERACT agents.Source: J Additional Q-Prediction Examples · p. 25
• Action-centric agents focus on perceptual representations of actions instead of assuming explicit object representations.Source: J Additional Q-Prediction Examples · p. 25
• This approach reduces the need for practitioners to define objects, which can be difficult and task- or embodiment-specific.Source: J Additional Q-Prediction Examples · p. 25

Real-world multi-camera setup

• Multiple Kinect-2 cameras produced noisy depth frames with many holes because of unresolved Time-of-Flight interference.Source: K Things that did not work · p. 26

Fourier features for positional embeddings

• Fourier features produced substantially worse performance than learned positional embeddings.Source: K Things that did not work · p. 26

Pre-trained vision features

• Using CLIP vision features was extremely slow because ResNet50 and decoder layers had to process four independent RGB frames.Source: K Things that did not work · p. 26

Upsampling at multiple self-attention layers

• Upsampling features at multiple self-attention layers did not work at all.Source: K Things that did not work · p. 26

Extreme rotation augmentation

• Pitch-and-roll augmentation still learned policies but substantially increased training time, with latent-size suitability remaining unclear.Source: K Things that did not work · p. 26

Using Adam instead of LAMB

• Adam led to worse performance than LAMB in both simulated and real-world experiments.Source: K Things that did not work · p. 26

Sampling-Based Motion Planner

• Sampling-based planning makes execution sensitive to randomized paths, which can matter for tasks requiring smooth motion such as pouring.Source: L Limitations and Risks · p. 26

Dynamic Manipulation

• Discrete-time actions are not easily applicable to dynamic tasks requiring real-time closed-loop maneuvering.Source: L Limitations and Risks · p. 26

Dexterous Manipulation

• Discretized actions are non-trivial for multi-fingered hands, especially under-actuated systems where feasibility and robustness remain unclear.Source: L Limitations and Risks · p. 26

Generlization to Novel Instances and Objects

• Randomized drawer textures and colors confuse the agent when training exposes it to only one drawer appearance.Source: L Limitations and Risks · pp. 26–27

Scope of Language Grounding

• PERACT’s verb-noun grounding is tied closely to demonstrated tasks rather than general meanings of verbs such as “cleaning.”Source: L Limitations and Risks · p. 27

Predicting Task Completion

• Current evaluations use an oracle to indicate task completion rather than a learned success classifier.Source: L Limitations and Risks · p. 27

History and Partial Observability

• Current-observation-only prediction makes history-dependent and partially observable tasks infeasible without additional completion or memory mechanisms.Source: L Limitations and Risks · p. 27

Multi-Task Optimization

• Uniform task sampling can hurt dissimilar tasks when optimization favors tasks sharing common elements.Source: L Limitations and Risks · p. 28