PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

Tsung-Wei Ke, Nikolaos Gkanatsios, Katerina Fragkiadaki

arXiv:2402.10885v3cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot manipulation requires policies that represent multimodal actions while reasoning robustly about 3D scenes. 3D Diffuser Actor combines tokenized 3D scene representations with conditional action diffusion, achieving state-of-the-art results on RLBench and CALVIN and real-world learning from few demonstrations.

  • Problem

    Existing work separately addresses multimodal action distributions with diffusion and viewpoint-robust spatial reasoning with 3D representations; the paper asks how to combine them for robot policies.

  • Method

    3D Diffuser Actor uses a relative 3D denoising transformer to fuse tokenized RGB-D scene features, language, proprioception, and noisy future end-effector trajectories.

  • Results

    3D Diffuser Actor sets new state-of-the-art results on RLBench and CALVIN, with 18.1% absolute multi-view and 13.1% single-view RLBench gains, plus real-world learning from a handful of demonstrations.

  • Takeaways & Limitations

    Combining spatially tokenized 3D representations with diffusion outperforms tested 2D, regression, classification, and holistic non-tokenized 3D alternatives.

  • Takeaways & Limitations

    The method requires camera calibration and depth, is evaluated on quasi-static tasks, and is slower on average than non-diffusion policies.

Abstract

from arXiv · show

Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

1 Introduction

3D Diffuser Actor combines diffusion-based action-distribution learning with 3D scene representations to address multimodal manipulation and improve spatial generalization. It achieves strong benchmark results across multi-view, single-view, and zero-shot settings while extending evaluation to real-world control.

  • Motivation: Robot manipulation is multimodal, so the paper frames policy learning as estimating action distributions conditioned on robot state.Human demonstrations can contain diverse task-optimal behaviors.
  • Motivation: Diffusion policies outperform deterministic and alternative action-distribution formulations, while prior methods commonly use oracle states or 2D images.Compared alternatives include variational autoencoders, mixture-of-Gaussians, classification-regression combinations, and energy-based objectives.
  • Motivation: 3D robot policies lift perspective-view features into a depth-based 3D workspace and have shown better generalization across camera viewpoints than 2D policies.The paper attributes this potential advantage to visual tokens and actions interacting in a common, viewpoint-robust 3D space.
  • Contribution: 3D Diffuser Actor combines diffusion with tokenized 3D scenes, language, and noisy end-effector trajectories, using relative-position 3D attention for translation equivariance.The model predicts translation and rotation errors for the end-effector.
  • Results: 18.1% absolute gain on RLBench multi-view setups and 13.1% on single-view setups establish new state-of-the-art results, while CALVIN improves by 9% relative.The method also learns multi-task real-world manipulation across 12 tasks from a handful of demonstrations.
  • Novelty: The work fills a gap by combining 3D robot policies with diffusion objectives, outperforming ChainedDiffuser and contrasting with holistic pooled 3D scene embeddings.The authors argue tokenized spatial disentanglement supports better generalization because holistic embeddings change with any scene change.

2 Related Work

Related work spans multimodal imitation-learning policies, robotics applications of diffusion models, and 2D versus 3D scene-to-action representations. 3D Diffuser Actor differs by directly predicting future manipulation trajectories and combining tokenized 3D scenes with diffusion.

  • Learning manipulation policies from demonstrations: Imitation-learning policies progressed from deterministic behavior cloning to classification, generative, variational, energy-based, and diffusion formulations for multimodal demonstrations.These approaches model action diversity with increasingly expressive distributions or objectives.
  • Diffusion models in robotics: Robotics diffusion models have modeled object arrangements, visual subgoals, offline reinforcement learning, and trajectories linking specified 3D keyposes.ChainedDiffuser replaces motion planners for keypose-to-keypose trajectory linking.
  • Diffusion models in robotics: Unlike ChainedDiffuser, 3D Diffuser Actor predicts the next end-effector keypose together with its linking trajectory, making the task harder than linking two given keyposes.It also differs from 3D Diffusion Policy, which uses 1D point-cloud embeddings.
  • 2D and 3D scene representations for robot manipulation: 2D image-to-action policies directly predict 6-DoF end-effector poses but require thousands of demonstrations to implicitly model 3D geometry and domain variation.The passage contrasts this with 3D scene-to-action policies.
  • 2D and 3D scene representations for robot manipulation: 3D Diffuser Actor uses a tokenized 3D scene representation with diffusion objectives rather than deterministic prediction, 3D action maps, classification, or regression.It is compared experimentally with both 2D diffusion policies and 3D policies.
  • 3D Diffuser Actor: The architecture overview describes conditional trajectory diffusion that iteratively denoises future actions and captures multiple behavioral modes in real-world demonstrations.The model conditions on visual observations, language, and proprioception.

3 Method

3D Diffuser Actor learns from demonstration trajectories by representing noisy future poses and RGB-D observations as spatially grounded 3D tokens. A relative denoising transformer predicts translation and rotation noise, and inference iteratively denoises complete trajectories.

  • Inputs and action representation: Training demonstrations pair visual observations and language instructions with end-effector actions decomposed into 3D location, rotation, and binary opening state.Observations consist of one or more posed RGB-D images.
  • Inputs and action representation: Rotations use the 6D representation to avoid quaternion discontinuities across environments.The trajectory notation includes location and rotation sequences over a temporal horizon.
  • Diffusion formulation: The policy is a conditional diffusion model that predicts whole future trajectories non-autoregressively through iterative denoising.A variance schedule controls how noise is added during the forward diffusion process.
  • 3D tokenization: At each diffusion step, the model converts noisy trajectory poses, visual observations, proprioception, and language into tokens, with each 3D token carrying a latent embedding and 3D position.Image features come from CLIP ResNet50, while depth and pixel coordinates determine 3D patch locations.
  • 3D Relative Denoising Transformer: Relative 3D self-attention fuses spatial tokens, language cross-attention adds instruction information, and MLP heads predict translation, rotation, and opening-state outputs.Relative positional information is encoded through rotary positional embeddings and a position-dependent matrix function.
  • Training and inference: Training adds sampled noise to ground-truth trajectories, reconstructs locations and rotations with L1 losses, and supervises opening state with binary cross-entropy.The objective weights location and rotation terms with cross-validated hyperparameters.
  • Training and inference: Inference samples Gaussian noise and progressively denoises it according to a schedule until obtaining a trajectory from the learned conditional distribution.Separate scaled-linear and square-cosine schedulers for location and rotation empirically improve performance.
  • Training and inference: Demonstrations are segmented at detected end-effector keyposes and resampled to a common length before training.Inference can execute a full trajectory to the next keypose or predict the next keypose for subsequent planning.

4 Experiments

Across RLBench, CALVIN, and real-world tasks, 3D Diffuser Actor is evaluated against 2D and 3D baselines, ablations, and alternative diffusion approaches. It achieves strong benchmark performance, while results attribute gains to 3D representations, diffusion-based action prediction, and relative attention, under stated sensing and task-setting limitations.

  • RLBench: 81.3% average success across 18 RLBench multi-view tasks, an absolute +18.1% improvement over Act3D.The largest gains occur on long-horizon, high-precision tasks with multiple action modes.
  • RLBench: +13.1% over Act3D and +46.7% over GNFactor on RLBench single-view evaluation.The comparison suggests 3D scene representation choices matter more than 3D feature completion in this setting.
  • Ablations: 3D Diffuser Actor outperforms its 2D counterpart and the absolute-attention ablation, supporting 3D representations and translation-equivariant relative attention.The absolute-attention variant still outperforms prior arts, while relative attention improves generalization.
  • CALVIN: The CALVIN evaluation compares 3D Diffuser Actor with 3D Diffusion Policy and ChainedDiffuser using language-annotated training data and final checkpoints across three seeds.The evaluation uses extracted keyposes and trajectory-level replanning rather than replanning after every individual action.
  • CALVIN: On CALVIN, 3D Diffuser Actor outperforms the state-of-the-art in zero-shot long-horizon evaluation across three random seeds.Longer termination horizons significantly increase performance, suggesting the model learns to retry after failures.
  • Real-world evaluation: With 15 demonstrations per task, 3D Diffuser Actor effectively learns multi-task manipulation in the real world.The demonstrations contain naturally occurring noise and multiple modes of human behavior, and success is evaluated over 10 episodes per task.
  • Limitations: The framework requires camera calibration and depth information, targets quasi-static tasks, and is slower on average than non-diffusion policies.The paper identifies extending the method to dynamic tasks and velocity control, and reducing diffusion inference steps, as future directions.

5 Conclusion

3D Diffuser Actor combines 3D scene representations with action diffusion for manipulation policies, achieving large-margin state-of-the-art results on RLBench and CALVIN and learning real-world control from few demonstrations.

  • 3D Diffuser Actor combines 3D scene representations and action diffusion for learning robot manipulation policies from demonstrations.
  • The method achieves state-of-the-art performance on RLBench and CALVIN by a large margin over existing 2D and 3D policies.
  • The policy also learns robot control in the real world from a handful of demonstrations.

A.1 Robustness to noisy depth information on RLBench

3D Diffuser Actor remains effective under noisy depth inputs, although performance declines as perturbation strength increases; RLBench failures mainly involve pose precision and instruction understanding.

  • Robustness to noisy depth information: 72.4% average success rate under mild depth perturbation, with only a 6% absolute performance drop.The evaluation uses the RLBench single-view setup across 10 tasks, three seeds, and 25 episodes per task.
  • Robustness to noisy depth information: 50.1% average success rate under strong depth perturbation, despite training exclusively with clean depth maps.
  • Failure cases: The main RLBench failure mode is imprecise end-effector pose prediction, especially in tasks requiring high precision.Examples include stacking blocks, opening a drawer, and turning a tap.
  • Failure cases: Instruction confusion is another major failure mode, particularly in scenes containing multiple objects.The policy may grasp an object different from the one specified by the instruction.
  • Failure cases: Additional failures arise from motion-planning failures and incomplete intermediate steps in long-horizon tasks.

A.3 RLBench tasks under multi-view setup

The RLBench multi-view suite comprises manipulation tasks varying object pose, appearance, and semantics, with success defined by completing the language-specified goal.

  • Tasks: The tasks cover opening, sliding, sweeping, removing, rotating, placing, screwing, inserting, pushing, stacking, and hanging objects.
  • Tasks: Several tasks require selecting objects or destinations by color, shape, or specified location.
  • Tasks: Other tasks require ordered button presses, peg placement, cup stacking, or hanging multiple mugs on a rack.

A.4 Real-world tasks

The real-world evaluation uses cluttered scenes with distractor objects and tasks spanning varied object selection, placement, insertion, opening, pressing, sorting, and stacking demands.

  • Task setting: Real-world tasks occur in cluttered scenes containing distractors that do not participate in the task.
  • Tasks: The task suite includes closing a box, placing a duck or mouse, inserting pegs, opening a pen, pressing a stapler, and putting grapes in a bowl.
  • Tasks: Additional tasks involve pressing a stapler and placing grapes in a bowl.
  • Tasks: Sorting rectangles, stacking same-shape blocks, stacking cups, and forming a triangle on a plate test multi-object arrangement.
  • Capabilities tested: The tasks examine multimodality, execution order, precision, and high noise or variance in keyposes.
  • Baselines: The comparison set includes 2D policies, 3D policies, ChainedDiffuser, and 3D Diffusion Policy.
  • Baselines: The 3D baselines use voxelization, workspace featurization, history attention, dense point representations, multi-view fusion, or neural-field reconstruction.
  • Baselines: 3D Diffusion Policy pools point-cloud representations into a holistic 1D feature vector, whereas ChainedDiffuser predicts intermediate trajectories toward target keyposes.

A.5.1 Re-training of 3D Diffusion Policy on CALVIN

The CALVIN comparison re-trains 3D Diffusion Policy with language conditioning and a specified point-cloud, training, and action-execution setup. The section also describes sparse keypose extraction based on gripper changes and significant motion changes.

  • CALVIN retraining: 3D Diffusion Policy was augmented with language conditioning through point-cloud-to-language cross-attention layers and a language encoder.The comparison used the original DP3 architecture rather than Simple DP3.
  • CALVIN retraining: CALVIN training used front and wrist RGB-D views, cropped depth regions, 1024 sampled points per view, batch size 5400, and 3000 epochs.The action horizon was 4, with 3 executed actions.
  • Keypose extraction: Keyposes are selected when the end-effector changes state or its velocity approaches zero, including pre-grasp poses and phase beginnings.The real-world procedure preserves these heuristics and reports keypose counts per task.
  • Keypose extraction: Motion-based keypose discovery identifies local acceleration maxima, retains prominent changes, and enforces sparse waypoint spacing.The implementation considers the top 20% of local maxima and uses a buffer size to maintain minimum distances between waypoints.

B Additional Method Details

The method represents RGB-D observations as 3D scene tokens and combines them with language, proprioception, and noisy pose trajectories through relative attention. Additional details contrast this tokenized architecture with other 3D policies and describe an enhanced language-conditioning variant.

  • Comparisons: PerAct uses voxelized RGB-D representations, whereas 3D Diffusion Policy average-pools point clouds into a holistic one-dimensional feature vector.Act3D instead lifts 2D feature maps into 3D and determines poses through iterative ghost-point classification.
  • Relative attention: Relative positional information is encoded with rotary positional embeddings so denoising attention remains translation invariant.The attention uses query, key, and value tokens together with a matrix function of relative point positions.
  • Language conditioning: The enhanced-language-conditioning variant of 3D Diffuser Actor achieves state-of-the-art results on CALVIN.The architecture variants are illustrated as standard and enhanced language-conditioning versions.
  • 3D Diffuser Actor: 3D Diffuser Actor lifts pretrained-backbone visual tokens into a 3D feature cloud and fuses them with language tokens, proprioception, and noisy position and rotation estimates.Its inputs include RGB-D views, a language instruction, current end-effector poses, noisy pose estimates, and the diffusion step.
  • 3D Diffuser Actor: Visual tokens attend to language, proprioception attends to visual tokens, and sampled visual, proprioceptive, and noisy pose tokens attend jointly.Farthest Point Sampling reduces the number of visual tokens; adaptive layer normalization and FiLM modulate attention.

B.5 Hyper-parameters for experiments

The experiments specify benchmark hyper-parameters and formulate the policy as conditional diffusion over end-effector pose trajectories. They also emphasize that rotation-noise scheduling materially affects the coverage of noised rotation representations.

  • Experimental settings: On CALVIN, overfitting led to lower test performance, motivating higher weight decay and fewer training epochs than on RLBench.These adjustments are described as benchmark-specific training choices.
  • Diffusion formulation: The diffusion process models end-effector translation and rotation trajectories by gradually adding noise according to a variance schedule and learning to predict that noise.The denoising network receives the noisy trajectory and diffusion step, while conditioning includes visual, language, and proprioceptive context.
  • Diffusion formulation: Sampling begins from Gaussian noise and progressively applies the denoising network according to a specified schedule to obtain a trajectory sample.The process iterates through the diffusion steps before producing the final sample.
  • Noise scheduling: The authors found that scheduler selection is critical because the squared linear scheduler covers the rotation space more completely than the scaled linear scheduler.The comparison visualizes clean and noised 6D rotations as two three-dimensional unit-length vectors.
Loading 2402.10885v3…