PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

Moritz Reuss, Ömer Erdinç Yağmurlu, Fabian Wenzel, Rudolf Lioutikov

arXiv:2407.05996v1cs.RO

TL;DR

Existing imitation-learning methods often require one goal modality and dense language annotation, whereas large-scale robot datasets are frequently only partially labeled. MDT addresses this with a multimodal diffusion transformer and two self-supervised objectives that align goal representations and encode future-state information. It performs strongly across CALVIN and LIBERO, including sparse-language settings, while achieving a 15% absolute gain on two CALVIN challenges.

  • Problem

    Imitation-learning methods often target either language or image goals, while large-scale demonstrations may have few language annotations and therefore provide limited data for language-conditioned training.

  • Method

    MDT combines a diffusion-based multimodal transformer with Contrastive Latent Alignment and Masked Generative Foresight to learn goal-conditioned representations from multimodal instructions.

  • Results

    15% absolute performance gain is reported on two CALVIN challenges, while MDT remains competitive on LIBERO with only 2% of demonstrations language-labeled.

  • Takeaways & Limitations

    MDT supports long-horizon manipulation from multimodal goals and sparsely annotated data across simulated and real-world settings.

  • Takeaways & Limitations

    MGF and CLA do not improve LIBERO-Long performance consistently, their impact varies by benchmark, and diffusion policies require multiple forward passes for action generation.

Abstract

from arXiv · show

This work introduces the Multimodal Diffusion Transformer (MDT), a novel diffusion policy framework, that excels at learning versatile behavior from multimodal goal specifications with few language annotations. MDT leverages a diffusion-based multimodal transformer backbone and two self-supervised auxiliary objectives to master long-horizon manipulation tasks based on multimodal goals. The vast majority of imitation learning methods only learn from individual goal modalities, e.g. either language or goal images. However, existing large-scale imitation learning datasets are only partially labeled with language annotations, which prohibits current methods from learning language conditioned behavior from these datasets. MDT addresses this challenge by introducing a latent goal-conditioned state representation that is simultaneously trained on multimodal goal instructions. This state representation aligns image and language based goal embeddings and encodes sufficient information to predict future states. The representation is trained via two self-supervised auxiliary objectives, enhancing the performance of the presented transformer backbone. MDT shows exceptional performance on 164 tasks provided by the challenging CALVIN and LIBERO benchmarks, including a LIBERO version that contains less than $2\%$ language annotations. Furthermore, MDT establishes a new record on the CALVIN manipulation challenge, demonstrating an absolute performance improvement of $15\%$ over prior state-of-the-art methods that require large-scale pretraining and contain $10\times$ more learnable parameters. MDT shows its ability to solve long-horizon manipulation from sparsely annotated data in both simulated and real-world environments. Demonstrations and Code are available at https://intuitive-robots.github.io/mdt_policy/.

I. INTRODUCTION

MDT targets multimodal robot instruction from sparsely annotated demonstrations, combining diffusion policies with self-supervised representation learning. It reports strong results across CALVIN and LIBERO, including a 15% absolute gain on two CALVIN challenges.

  • Motivation: Current imitation-learning methods commonly specialize in either language or visual goals, while large uncurated datasets contain too few language annotations for fully language-conditioned training.This limits training to fully labeled datasets, whose collection is time-consuming and labor-intensive.
  • Approach: MDT learns versatile behavior from language and image goals simultaneously, including when only a small fraction of demonstrations have language labels.The approach is designed for multimodal goals in partially annotated datasets.
  • Approach: MDT combines a multimodal transformer encoder-decoder with diffusion-based action denoising and latent goal-conditioned representation learning.The architecture processes image observations and multimodal goals before denoising future action sequences.
  • Approach: Two self-supervised objectives, Masked Generative Foresight and Contrastive Latent Alignment, improve multimodal-goal learning without additional inference costs.They encourage latent features that align goals across modalities and reconstruct partially masked future frames.
  • Evaluation context: The study evaluates MDT and its objectives across more than 184 tasks and compares them with diffusion, transformer, and multimodal-goal methods.The related work includes diffusion policies, self-supervised robotics representations, and behavior generation from multiple goal modalities.

III. METHOD

MDT learns goal-conditioned action sequences from task-agnostic play trajectories using a continuous-time diffusion policy. Its formulation conditions actions on current state embeddings and latent goals represented by either future images or language instructions.

  • Problem formulation: MDT predicts length-k action sequences conditioned on the current state embedding and a latent goal.The latent goal can encode either a future goal image or a free-form language instruction.
  • Problem formulation: The play dataset contains state-action windows extracted from task-agnostic trajectories, with available language annotations and sampled future goal images.Future goal images are sampled with offsets j ∈ [20, 50] from a geometric distribution with probability 0.1.
  • Problem formulation: The policy must represent multiple trajectories that converge to the same goal because human behavior is diverse.This motivates a versatile conditional policy for learning from play data.
  • Score-based diffusion policy: MDT uses a continuous-time diffusion formulation in which Gaussian noise perturbs action sequences across a scheduled noise level.The policy uses σ_t = t and models the score function over the noisy action sequence.
  • Score-based diffusion policy: A trainable score network learns to denoise randomly perturbed action sequences during training.During rollout, the learned score model is inserted into the reverse SDE to iteratively generate actions.
  • Score-based diffusion policy: Setting β_t = 0 yields a deterministic inverse process that enables fast sampling in a few denoising steps.This removes additional noise injection from the inverse process.

C. Model Architecture

MDT combines a multimodal transformer encoder-decoder with auxiliary foresight objectives. The encoder represents current observations and goals, while the decoder denoises future actions and MGF trains the latent representation to predict masked future visual content.

  • Model architecture: MDT encodes current image observations and multimodal goals into latent representation tokens, then denoises future action sequences with a diffusion decoder.Cross-attention injects encoder conditioning into every decoder layer.
  • Model architecture: The architecture supports multiple image views and embeds goal images and language annotations with frozen CLIP models.The default MDT uses trainable ResNet-18 encoders, while MDT-V uses frozen Voltron embeddings and a Perceiver-Resampler.
  • Masked Generative Foresight: MGF conditions a future-image decoder on the latent state representation derived from the current observation and goal.The decoder receives encoded future-image patches and mask tokens to reconstruct occluded patches.
  • Evaluation environments: The model is evaluated across CALVIN, LIBERO, and real-robot environments spanning varied tasks, scenes, and partially labeled play data.The environments include four CALVIN setups, five LIBERO task suites, and toy-kitchen experiments.
  • Masked Generative Foresight: MGF reconstructs masked patches from a future state rather than reconstructing the current image or full video.Only masked patches contribute to the reconstruction loss.
  • Masked Generative Foresight: MGF is designed to add foresight to latent embeddings and can be applied beyond MDT to transformer policies.The method differs from masking approaches that focus on robust representations of the current state.

E. Aligning Latent Goal-Conditioned Representations

MDT aligns image- and language-conditioned goal representations because static CLIP embeddings do not fully capture the spatial dynamics of robotic goals. It uses contrastive latent alignment alongside score matching and MGF, and reports strong CALVIN performance.

  • Motivation: Static CLIP embeddings can struggle with spatial relationships and dynamics that are central to robotic goal specifications.This motivates learning alignment within the state and goal-conditioned policy representation.
  • Contrastive Latent Alignment: CLA aligns image- and language-conditioned state embeddings by reducing each modality to a normalized vector and applying an InfoNCE objective.The image and language representations are denoted z_i^o and z_i^l, respectively.
  • CALVIN results: MDT and MDT-V significantly outperform reported baselines on both CALVIN ABCD→D and D→D challenges.The comparison uses average rollout length for solving five instructions consecutively across 1000 chains, averaged over three seeds.
  • Training objective: The overall training objective combines score matching, MGF, and CLA losses.The auxiliary-loss weights are α = 0.1 and β = 0.1 in most experiment settings.

IV. EVALUATION

The evaluation tests whether MDT learns long-horizon manipulation from play data with few language annotations, improves through MGF and CLA, transfers MGF to another transformer policy, and works in real-world settings. On CALVIN, MDT-V reaches a new record while using fewer parameters and no large-scale pretraining.

  • Evaluation scope: MDT is evaluated on CALVIN, LIBERO, and a real-world play setting against several state-of-the-art language-conditioned policies.The experiments address long-horizon learning, auxiliary-loss benefits, transfer to other transformer policies, and partially labeled real-world manipulation.
  • Baselines: MDT is compared with HULC, LAD, Distill-D, MT-ACT, and RoboFlamingo under matched language and image goal models where applicable.RoboFlamingo is a 3-billion-parameter vision-language model pretrained on large internet-scale image-text data.
  • CALVIN: 4.52 average rollout length: MDT-V sets a new CALVIN record, improving 10% over RoboFlamingo.On CALVIN ABCD→D, MDT-V also uses less than 10% of the trainable parameters and requires no large-scale pretraining.
  • CALVIN: 20% average rollout-length improvement: MDT-V outperforms the second-best baseline on the scaled-down CALVIN D→D benchmark.The D→D setting uses only 6 hours of play data, compared with the full ABCD→D training setting.

B. Evaluation on LIBERO

LIBERO evaluates MDT across five task suites under sparse language supervision, alongside a real-world play-kitchen study using partially labeled long-horizon demonstrations. MDT outperforms fully labeled or U-Net-based comparators on LIBERO and improves further with auxiliary objectives, while real-world results show gains over MT-ACT.

  • LIBERO: MDT evaluates over 130 LIBERO tasks across five suites with only 2% of demonstrations carrying language labels.Evaluation uses static and wrist-mounted cameras, 20 rollouts per task, and averages results over 3 seeds.
  • LIBERO results: MDT outperforms the fully labeled Oracle-BC baseline and beats the fully language-labeled Transformer Baseline in three of four LIBERO challenges.It also surpasses the U-Net-based Distill-D policy in all tests, even without auxiliary objectives.
  • LIBERO results: 8.5% average improvement: the auxiliary objectives raise MDT performance across the five LIBERO task suites.The cited results report improvement in four of five suites.
  • Real-world setup: The real-world play dataset contains about 4.5 hours of teleoperated data spanning 20 tasks, with approximately 20% of the dataset labeled.Demonstrations last roughly 30 to more than 450 seconds and contain between 5 and 20 tasks.
  • Real-world results: 0.58 success rate: MDT with auxiliary objectives exceeds MDT without them at 0.51 and MT-ACT at 0.25 on single-task real-world evaluation.Results are averaged over 5 rollouts for each of 20 tasks.
  • Real-world results: Long-horizon real-world evaluation requires policies to chain several behaviors, receiving only the next goal after completing the previous task.The study compares MDT and its auxiliary-objective variant with MT-ACT.

D. Evaluation of Auxiliary Losses

The auxiliary-loss study measures MGF and CLA on CALVIN, LIBERO, transfer to MT-ACT, and real-world play-kitchen rollouts. Both objectives generally improve performance, although their effects vary by benchmark and task structure.

  • CALVIN ablations: Over 25% average rollout-length improvement: adding MGF enhances MDT on CALVIN ABCD→D.Without both auxiliary losses, MDT-V still surpasses all baselines with an average rollout length of 4.12.
  • LIBERO ablations: More than 8.5% average increase: MGF and CLA improve MDT success rates in four of five LIBERO task suites.The study reports a synergistic effect when both losses are applied together; CLA has greater impact than MGF overall, especially on LIBERO-90.
  • Scope of auxiliary benefits: LIBERO-Long does not appear to benefit from MGF or CLA, possibly because its demonstrations contain several subtasks but only one high-level description.The paper attributes the lack of notable benefit to missing sub-goals.
  • Transfer to other policies: 44% average CALVIN-performance boost: adding MGF improves MT-ACT without changing its model or hyperparameters.This supports applying MGF beyond MDT to another transformer-based policy.
  • Real-world evaluation: The auxiliary objectives substantially increase MDT’s average rollout length on the long-horizon real-world play-kitchen dataset.Table IV averages performance over 4 instruction chains with 4 rollouts each and compares MDT with MT-ACT.

E. Additional Ablation Studies

The ablations examine auxiliary objectives, goal encoders, masking, transformer architecture, and pretraining, while the authors note benchmark-dependent effects and remaining limitations.

  • Choice of Goal Image Encoder: MDT with ResNet goal encoders matches frozen CLIP embeddings at 3.34 average rollout length without auxiliary tasks.The passage introduces a comparison of auxiliary objectives with the ResNet variant but does not report its completed result.
  • Pretraining with MGF and CLA: 100% and 25% performance increases result from auxiliary-objective pretraining with 5 and 20 demonstrations, respectively, on LIBERO-Long.Pretraining uses video data without robot actions; auxiliary objectives also improve performance when MDT is trained from scratch.
  • Masked Generative Foresight: A 0.75 masking ratio achieves the best average performance for Masked Generative Foresight and becomes the default setting.The tested masking ratios range from 0.5 to 1.
  • Transformer Architecture: 4.18 average rollout length for complete MDT exceeds 3.58 without adaLN conditioning and 1.41 without the transformer encoder.The comparisons are performed on CALVIN ABCD→D without auxiliary objectives.
  • Limitations: MGF and CLA do not improve LIBERO-Long, and their performance impact varies across benchmarks.CLA has a higher impact on LIBERO, whereas MGF has the opposite pattern on CALVIN.
  • Conclusion: MDT is tested across 184 tasks and achieves an average 15% absolute improvement over prior state of the art on CALVIN.The work also studies auxiliary objectives and reports experiments in simulated and real-world settings.

APPENDIX

The appendix specifies MDT’s continuous-time diffusion implementation, sampling strategy, and comparison-policy configurations, including shared diffusion hyperparameters for fair evaluation.

  • MDT diffusion implementation: MDT uses Karras-style preconditioning with skip connections and noise-level-conditioned layers in its score-matching process.The conditioning is based on the current noise level σ_t.
  • MDT diffusion implementation: Denoised action sequences are generated during rollouts with a DDIM reverse-diffusion process.The training and rollout procedures are summarized in separate algorithms.
  • MDT diffusion implementation: The denoiser combines skip-connected noisy actions with a transformer output through noise-dependent functions.The displayed formulation includes c_skip(σ_t), c_out(σ_t), c_in(σ_t), and c_noise(σ_t).
  • Sampling strategy: Geometric future-window sampling with probability p = 0.1 is used for CALVIN and real-robot experiments.Random sampling caused small performance drops, while key-state-based goal sampling did not work well.
  • Comparison policies: MT-ACT uses default ResNets as vision encoders because FiLM-conditioned and pretrained Voltron encoders performed poorly for the tested settings.Voltron reduced CALVIN ABCD→D performance by over 20% for MT-ACT.
  • Comparison policies: Distill-D and MDT use the same diffusion hyperparameters to support fair comparison on CALVIN and LIBERO.Distill-D is based on the Diffusion-policy implementation with the continuous-time diffusion variant.
  • Comparison policies: Distill-D contains 296.6 million trainable 1D-CNN parameters plus 22.4 million parameters for two ResNet-18 encoders.Pretrained Voltron embeddings did not improve Distill-D performance.

D. CALVIN Experiment Details

The CALVIN experiments follow a standardized evaluation protocol using static and wrist-camera observations, fixed training budgets, and ablations of masking and auxiliary objectives.

  • Evaluation protocol: CALVIN policies use static-camera and wrist-camera images with random-shift augmentation and resolutions of 224 × 224 and 84 × 84 pixels.The protocol follows the evaluation procedure used for consistent comparisons with prior work.
  • Real-robot setup: The real-robot kitchen setup contains a play kitchen, multiple objects, cameras, and a second robot used for data collection and teleoperation.The figure presents the environment and equipment used for the real-robot experiments.
  • Ablation coverage: The CALVIN ablations evaluate masking rates, prediction horizons, auxiliary objectives, transformer variants, and ResNet goal encoders.The appendix tables report these comparisons on CALVIN ABCD→D and related settings.
  • Training setup: Training runs use 20,000 steps on the smaller dataset and 30,000 steps on the complete dataset.Longer training did not improve performance and was avoided because full-dataset training is computationally demanding.

E. LIBERO Experiment Details

The LIBERO experiments use five task suites with 50 demonstrations per task and simulate sparse language supervision by labeling only one demonstration per task.

  • Dataset and supervision: Each LIBERO task has 50 demonstrations, divided into one language-annotated demonstration and 49 demonstrations without labels.Goal images are generated from the final state of each rollout, using the default end-effector action space.
  • Task suites: The Spatial suite tests distinguishing identical bowls through their spatial relationships to surrounding objects.Tasks involve placing a bowl on a plate among a constant set of objects.
  • Task suites: The Goal suite varies the ultimate objective while keeping the object set and spatial relationships fixed.The suite tests adaptation of motions and behaviors to different goals.
  • Task suites: The Object suite requires recognizing and manipulating a unique object in each task.The stated task is picking and placing the object.
  • Task suites: The Long suite evaluates long-horizon planning and performance over extended task durations.Its challenge is maintaining adaptability throughout longer executions.
  • Task suites: The 90 suite contains 90 diverse short-horizon tasks across five settings.

F. Real Robot Experiments

The real-robot experiments evaluate MDT on multimodal and long-horizon manipulation using teleoperated play data, sparse language annotations, goal images, and text instructions. The experiments also reveal failures on ambiguous object-placement tasks and precise door or tray actions.

  • Experimental setup: The robot setup contains a toy kitchen with 20 diverse tasks and evaluates policies from two static camera perspectives.The environment includes a microwave, oven, cooler, sink, banana, pot, and toaster with toast.
  • Data collection: Teleoperated play data were collected at 6 Hz with proprioceptive signals and two camera images, using normalized desired joint positions as actions.The data came from a leader-follower robot setup.
  • Data collection: Only 360 short-horizon segments received language annotations, with GPT-4 generating varied instructions for each task.The labeled segments were sampled from the uncurated play dataset and described by a human before instruction diversification.
  • Training and evaluation: Policies were trained for about 24 hours on four GPUs over roughly 100 epochs, selecting checkpoints primarily by validation loss.MT-ACT was instead evaluated using the last epoch because prior benchmark experience suggested later improvement despite rising validation loss.
  • Training and evaluation: Evaluation used 10 goal images and 10 text instructions per task, five trials from unseen noisy starts, and four long-horizon chains of five or six tasks.The long-horizon chains included Random, Open Close All, Stovetop Sink, and Oven sequences.
  • Results and failure cases: Policies often misunderstood similar banana- and pot-placement goals and missed door-closing or oven-tray actions by a few millimeters.The authors hypothesize that sparse labels, similar states, tight actions, varying openness, and quick demonstrations contributed to these failures.
  • Results and failure cases: In real-robot goal-image rollouts, 4 out of 6 tasks succeeded, while the robot failed to open the oven door and opened the ice box instead.The result is reported for the illustrated goal-image-conditioned rollouts.
Loading 2407.05996v1…