PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

Moritz Reuss, Ömer Erdinç Yağmurlu, Fabian Wenzel, Rudolf Lioutikov

arXiv:2407.05996v1cs.RO

TL;DR

Existing imitation-learning methods often require one goal modality and dense language annotation, whereas large-scale robot datasets are frequently only partially labeled. MDT addresses this with a multimodal diffusion transformer and two self-supervised objectives that align goal representations and encode future-state information. It performs strongly across CALVIN and LIBERO, including sparse-language settings, while achieving a 15% absolute gain on two CALVIN challenges.

  • Problem

    Imitation-learning methods often target either language or image goals, while large-scale demonstrations may have few language annotations and therefore provide limited data for language-conditioned training.

    Source: §I · p. 1

  • Method

    MDT combines a diffusion-based multimodal transformer with Contrastive Latent Alignment and Masked Generative Foresight to learn goal-conditioned representations from multimodal instructions.

    Source: §I · p. 1, §III · p. 3, E. Aligning Latent Goal-Conditioned… · p. 5

  • Results

    15% absolute performance gain is reported on two CALVIN challenges, while MDT remains competitive on LIBERO with only 2% of demonstrations language-labeled.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    MDT supports long-horizon manipulation from multimodal goals and sparsely annotated data across simulated and real-world settings.

    Source: §I · p. 2, §V · p. 10

    MGF and CLA do not improve LIBERO-Long performance consistently, their impact varies by benchmark, and diffusion policies require multiple forward passes for action generation.

    Source: F. Limitations · p. 10

Abstract

from arXiv · show

This work introduces the Multimodal Diffusion Transformer (MDT), a novel diffusion policy framework, that excels at learning versatile behavior from multimodal goal specifications with few language annotations. MDT leverages a diffusion-based multimodal transformer backbone and two self-supervised auxiliary objectives to master long-horizon manipulation tasks based on multimodal goals. The vast majority of imitation learning methods only learn from individual goal modalities, e.g. either language or goal images. However, existing large-scale imitation learning datasets are only partially labeled with language annotations, which prohibits current methods from learning language conditioned behavior from these datasets. MDT addresses this challenge by introducing a latent goal-conditioned state representation that is simultaneously trained on multimodal goal instructions. This state representation aligns image and language based goal embeddings and encodes sufficient information to predict future states. The representation is trained via two self-supervised auxiliary objectives, enhancing the performance of the presented transformer backbone. MDT shows exceptional performance on 164 tasks provided by the challenging CALVIN and LIBERO benchmarks, including a LIBERO version that contains less than $2\%$ language annotations. Furthermore, MDT establishes a new record on the CALVIN manipulation challenge, demonstrating an absolute performance improvement of $15\%$ over prior state-of-the-art methods that require large-scale pretraining and contain $10\times$ more learnable parameters. MDT shows its ability to solve long-horizon manipulation from sparsely annotated data in both simulated and real-world environments. Demonstrations and Code are available at https://intuitive-robots.github.io/mdt_policy/.

I. INTRODUCTION

MDT targets multimodal robot instruction from sparsely annotated demonstrations, combining diffusion policies with self-supervised representation learning. It reports strong results across CALVIN and LIBERO, including a 15% absolute gain on two CALVIN challenges.

Motivation

  • Current imitation-learning methods commonly specialize in either language or visual goals, while large uncurated datasets contain too few language annotations for fully language-conditioned training.This limits training to fully labeled datasets, whose collection is time-consuming and labor-intensive.Source: §I · p. 1

Approach

  • MDT learns versatile behavior from language and image goals simultaneously, including when only a small fraction of demonstrations have language labels.The approach is designed for multimodal goals in partially annotated datasets.Source: §I · pp. 1–2
  • MDT combines a multimodal transformer encoder-decoder with diffusion-based action denoising and latent goal-conditioned representation learning.The architecture processes image observations and multimodal goals before denoising future action sequences.Source: Figure 1 · p. 2
  • Two self-supervised objectives, Masked Generative Foresight and Contrastive Latent Alignment, improve multimodal-goal learning without additional inference costs.They encourage latent features that align goals across modalities and reconstruct partially masked future frames.Source: §I · pp. 1–2

Evaluation context

  • The study evaluates MDT and its objectives across more than 184 tasks and compares them with diffusion, transformer, and multimodal-goal methods.The related work includes diffusion policies, self-supervised robotics representations, and behavior generation from multiple goal modalities.Source: §I · p. 2, §II · pp. 2–3

III. METHOD

MDT learns goal-conditioned action sequences from task-agnostic play trajectories using a continuous-time diffusion policy. Its formulation conditions actions on current state embeddings and latent goals represented by either future images or language instructions.

Problem formulation

  • MDT predicts length-k action sequences conditioned on the current state embedding and a latent goal.The latent goal can encode either a future goal image or a free-form language instruction.Source: A. Problem Formulation · p. 3
  • The play dataset contains state-action windows extracted from task-agnostic trajectories, with available language annotations and sampled future goal images.Future goal images are sampled with offsets j ∈ [20, 50] from a geometric distribution with probability 0.1.Source: A. Problem Formulation · p. 3
  • The policy must represent multiple trajectories that converge to the same goal because human behavior is diverse.This motivates a versatile conditional policy for learning from play data.Source: A. Problem Formulation · p. 3

Score-based diffusion policy

  • MDT uses a continuous-time diffusion formulation in which Gaussian noise perturbs action sequences across a scheduled noise level.The policy uses σ_t = t and models the score function over the noisy action sequence.Source: B. Score-based Diffusion Policy · pp. 3–4
  • A trainable score network learns to denoise randomly perturbed action sequences during training.During rollout, the learned score model is inserted into the reverse SDE to iteratively generate actions.Source: B. Score-based Diffusion Policy · p. 4
  • Setting β_t = 0 yields a deterministic inverse process that enables fast sampling in a few denoising steps.This removes additional noise injection from the inverse process.Source: B. Score-based Diffusion Policy · p. 4

C. Model Architecture

MDT combines a multimodal transformer encoder-decoder with auxiliary foresight objectives. The encoder represents current observations and goals, while the decoder denoises future actions and MGF trains the latent representation to predict masked future visual content.

Model architecture

  • MDT encodes current image observations and multimodal goals into latent representation tokens, then denoises future action sequences with a diffusion decoder.Cross-attention injects encoder conditioning into every decoder layer.Source: C. Model Architecture · p. 4
  • The architecture supports multiple image views and embeds goal images and language annotations with frozen CLIP models.The default MDT uses trainable ResNet-18 encoders, while MDT-V uses frozen Voltron embeddings and a Perceiver-Resampler.Source: C. Model Architecture · p. 4

Masked Generative Foresight

  • MGF conditions a future-image decoder on the latent state representation derived from the current observation and goal.The decoder receives encoded future-image patches and mask tokens to reconstruct occluded patches.Source: Figure 2 · p. 4, D. Masked Generative Foresight · p. 5

Evaluation environments

  • The model is evaluated across CALVIN, LIBERO, and real-robot environments spanning varied tasks, scenes, and partially labeled play data.The environments include four CALVIN setups, five LIBERO task suites, and toy-kitchen experiments.Source: Figure 3 · p. 5

Masked Generative Foresight

  • MGF reconstructs masked patches from a future state rather than reconstructing the current image or full video.Only masked patches contribute to the reconstruction loss.Source: D. Masked Generative Foresight · p. 5
  • MGF is designed to add foresight to latent embeddings and can be applied beyond MDT to transformer policies.The method differs from masking approaches that focus on robust representations of the current state.Source: D. Masked Generative Foresight · p. 5

E. Aligning Latent Goal-Conditioned Representations

MDT aligns image- and language-conditioned goal representations because static CLIP embeddings do not fully capture the spatial dynamics of robotic goals. It uses contrastive latent alignment alongside score matching and MGF, and reports strong CALVIN performance.

Motivation

  • Static CLIP embeddings can struggle with spatial relationships and dynamics that are central to robotic goal specifications.This motivates learning alignment within the state and goal-conditioned policy representation.Source: E. Aligning Latent Goal-Conditioned… · p. 5

Contrastive Latent Alignment

  • CLA aligns image- and language-conditioned state embeddings by reducing each modality to a normalized vector and applying an InfoNCE objective.The image and language representations are denoted z_i^o and z_i^l, respectively.Source: E. Aligning Latent Goal-Conditioned… · p. 5

CALVIN results

  • MDT and MDT-V significantly outperform reported baselines on both CALVIN ABCD→D and D→D challenges.The comparison uses average rollout length for solving five instructions consecutively across 1000 chains, averaged over three seeds.Source: Table I · p. 6

Training objective

IV. EVALUATION

The evaluation tests whether MDT learns long-horizon manipulation from play data with few language annotations, improves through MGF and CLA, transfers MGF to another transformer policy, and works in real-world settings. On CALVIN, MDT-V reaches a new record while using fewer parameters and no large-scale pretraining.

Evaluation scope

  • MDT is evaluated on CALVIN, LIBERO, and a real-world play setting against several state-of-the-art language-conditioned policies.The experiments address long-horizon learning, auxiliary-loss benefits, transfer to other transformer policies, and partially labeled real-world manipulation.Source: §IV · p. 6

Baselines

  • MDT is compared with HULC, LAD, Distill-D, MT-ACT, and RoboFlamingo under matched language and image goal models where applicable.RoboFlamingo is a 3-billion-parameter vision-language model pretrained on large internet-scale image-text data.Source: A. Evaluation on CALVIN · p. 6

CALVIN

  • 4.52 average rollout length: MDT-V sets a new CALVIN record, improving 10% over RoboFlamingo.On CALVIN ABCD→D, MDT-V also uses less than 10% of the trainable parameters and requires no large-scale pretraining.Source: A. Evaluation on CALVIN · p. 6
  • 20% average rollout-length improvement: MDT-V outperforms the second-best baseline on the scaled-down CALVIN D→D benchmark.The D→D setting uses only 6 hours of play data, compared with the full ABCD→D training setting.Source: A. Evaluation on CALVIN · p. 6

B. Evaluation on LIBERO

LIBERO evaluates MDT across five task suites under sparse language supervision, alongside a real-world play-kitchen study using partially labeled long-horizon demonstrations. MDT outperforms fully labeled or U-Net-based comparators on LIBERO and improves further with auxiliary objectives, while real-world results show gains over MT-ACT.

LIBERO

  • MDT evaluates over 130 LIBERO tasks across five suites with only 2% of demonstrations carrying language labels.Evaluation uses static and wrist-mounted cameras, 20 rollouts per task, and averages results over 3 seeds.Source: B. Evaluation on LIBERO · p. 7

LIBERO results

  • MDT outperforms the fully labeled Oracle-BC baseline and beats the fully language-labeled Transformer Baseline in three of four LIBERO challenges.It also surpasses the U-Net-based Distill-D policy in all tests, even without auxiliary objectives.Source: B. Evaluation on LIBERO · p. 7
  • 8.5% average improvement: the auxiliary objectives raise MDT performance across the five LIBERO task suites.The cited results report improvement in four of five suites.Source: B. Evaluation on LIBERO · p. 7

Real-world setup

  • The real-world play dataset contains about 4.5 hours of teleoperated data spanning 20 tasks, with approximately 20% of the dataset labeled.Demonstrations last roughly 30 to more than 450 seconds and contain between 5 and 20 tasks.Source: C. Real Robot Experiments · p. 7

Real-world results

  • 0.58 success rate: MDT with auxiliary objectives exceeds MDT without them at 0.51 and MT-ACT at 0.25 on single-task real-world evaluation.Results are averaged over 5 rollouts for each of 20 tasks.Source: C. Real Robot Experiments · p. 8
  • Long-horizon real-world evaluation requires policies to chain several behaviors, receiving only the next goal after completing the previous task.The study compares MDT and its auxiliary-objective variant with MT-ACT.Source: C. Real Robot Experiments · p. 8

D. Evaluation of Auxiliary Losses

The auxiliary-loss study measures MGF and CLA on CALVIN, LIBERO, transfer to MT-ACT, and real-world play-kitchen rollouts. Both objectives generally improve performance, although their effects vary by benchmark and task structure.

CALVIN ablations

  • Over 25% average rollout-length improvement: adding MGF enhances MDT on CALVIN ABCD→D.Without both auxiliary losses, MDT-V still surpasses all baselines with an average rollout length of 4.12.Source: D. Evaluation of Auxiliary Losses · p. 8

LIBERO ablations

  • More than 8.5% average increase: MGF and CLA improve MDT success rates in four of five LIBERO task suites.The study reports a synergistic effect when both losses are applied together; CLA has greater impact than MGF overall, especially on LIBERO-90.Source: D. Evaluation of Auxiliary Losses · p. 8

Scope of auxiliary benefits

  • LIBERO-Long does not appear to benefit from MGF or CLA, possibly because its demonstrations contain several subtasks but only one high-level description.The paper attributes the lack of notable benefit to missing sub-goals.Source: D. Evaluation of Auxiliary Losses · p. 8

Transfer to other policies

  • 44% average CALVIN-performance boost: adding MGF improves MT-ACT without changing its model or hyperparameters.This supports applying MGF beyond MDT to another transformer-based policy.Source: D. Evaluation of Auxiliary Losses · p. 8

Real-world evaluation

  • The auxiliary objectives substantially increase MDT’s average rollout length on the long-horizon real-world play-kitchen dataset.Table IV averages performance over 4 instruction chains with 4 rollouts each and compares MDT with MT-ACT.Source: Table IV · p. 9

E. Additional Ablation Studies

The ablations examine auxiliary objectives, goal encoders, masking, transformer architecture, and pretraining, while the authors note benchmark-dependent effects and remaining limitations.

Choice of Goal Image Encoder

  • MDT with ResNet goal encoders matches frozen CLIP embeddings at 3.34 average rollout length without auxiliary tasks.The passage introduces a comparison of auxiliary objectives with the ResNet variant but does not report its completed result.Source: E. Additional Ablation Studies · p. 9

Pretraining with MGF and CLA

  • 100% and 25% performance increases result from auxiliary-objective pretraining with 5 and 20 demonstrations, respectively, on LIBERO-Long.Pretraining uses video data without robot actions; auxiliary objectives also improve performance when MDT is trained from scratch.Source: E. Additional Ablation Studies · p. 9

Masked Generative Foresight

  • A 0.75 masking ratio achieves the best average performance for Masked Generative Foresight and becomes the default setting.The tested masking ratios range from 0.5 to 1.Source: E. Additional Ablation Studies · p. 9

Transformer Architecture

  • 4.18 average rollout length for complete MDT exceeds 3.58 without adaLN conditioning and 1.41 without the transformer encoder.The comparisons are performed on CALVIN ABCD→D without auxiliary objectives.Source: E. Additional Ablation Studies · p. 10

Limitations

  • MGF and CLA do not improve LIBERO-Long, and their performance impact varies across benchmarks.CLA has a higher impact on LIBERO, whereas MGF has the opposite pattern on CALVIN.Source: F. Limitations · p. 10

Conclusion

  • MDT is tested across 184 tasks and achieves an average 15% absolute improvement over prior state of the art on CALVIN.The work also studies auxiliary objectives and reports experiments in simulated and real-world settings.Source: §V · p. 10

APPENDIX

The appendix specifies MDT’s continuous-time diffusion implementation, sampling strategy, and comparison-policy configurations, including shared diffusion hyperparameters for fair evaluation.

MDT diffusion implementation

  • MDT uses Karras-style preconditioning with skip connections and noise-level-conditioned layers in its score-matching process.The conditioning is based on the current noise level σ_t.Source: A. Diffusion Model Training and… · p. 14
  • Denoised action sequences are generated during rollouts with a DDIM reverse-diffusion process.The training and rollout procedures are summarized in separate algorithms.Source: A. Diffusion Model Training and… · p. 14
  • The denoiser combines skip-connected noisy actions with a transformer output through noise-dependent functions.The displayed formulation includes c_skip(σ_t), c_out(σ_t), c_in(σ_t), and c_noise(σ_t).Source: A. Diffusion Model Training and… · p. 14

Sampling strategy

  • Geometric future-window sampling with probability p = 0.1 is used for CALVIN and real-robot experiments.Random sampling caused small performance drops, while key-state-based goal sampling did not work well.Source: B. Goal Sampling Strategy · p. 14

Comparison policies

  • MT-ACT uses default ResNets as vision encoders because FiLM-conditioned and pretrained Voltron encoders performed poorly for the tested settings.Voltron reduced CALVIN ABCD→D performance by over 20% for MT-ACT.Source: C. Baseline Implementations · p. 14
  • Distill-D and MDT use the same diffusion hyperparameters to support fair comparison on CALVIN and LIBERO.Distill-D is based on the Diffusion-policy implementation with the continuous-time diffusion variant.Source: Table VI · p. 14
  • Distill-D contains 296.6 million trainable 1D-CNN parameters plus 22.4 million parameters for two ResNet-18 encoders.Pretrained Voltron embeddings did not improve Distill-D performance.Source: C. Baseline Implementations · p. 14

D. CALVIN Experiment Details

The CALVIN experiments follow a standardized evaluation protocol using static and wrist-camera observations, fixed training budgets, and ablations of masking and auxiliary objectives.

Evaluation protocol

  • CALVIN policies use static-camera and wrist-camera images with random-shift augmentation and resolutions of 224 × 224 and 84 × 84 pixels.The protocol follows the evaluation procedure used for consistent comparisons with prior work.Source: D. CALVIN Experiment Details · p. 14

Real-robot setup

  • The real-robot kitchen setup contains a play kitchen, multiple objects, cameras, and a second robot used for data collection and teleoperation.The figure presents the environment and equipment used for the real-robot experiments.Source: Figure 6 · p. 15

Ablation coverage

  • The CALVIN ablations evaluate masking rates, prediction horizons, auxiliary objectives, transformer variants, and ResNet goal encoders.The appendix tables report these comparisons on CALVIN ABCD→D and related settings.Source: Table VII · p. 15, Table VIII · p. 15, Table IX · p. 16

Training setup

  • Training runs use 20,000 steps on the smaller dataset and 30,000 steps on the complete dataset.Longer training did not improve performance and was avoided because full-dataset training is computationally demanding.Source: D. CALVIN Experiment Details · p. 15

E. LIBERO Experiment Details

The LIBERO experiments use five task suites with 50 demonstrations per task and simulate sparse language supervision by labeling only one demonstration per task.

Dataset and supervision

  • Each LIBERO task has 50 demonstrations, divided into one language-annotated demonstration and 49 demonstrations without labels.Goal images are generated from the final state of each rollout, using the default end-effector action space.Source: E. LIBERO Experiment Details · p. 16

Task suites

  • The Spatial suite tests distinguishing identical bowls through their spatial relationships to surrounding objects.Tasks involve placing a bowl on a plate among a constant set of objects.Source: E. LIBERO Experiment Details · p. 16
  • The Goal suite varies the ultimate objective while keeping the object set and spatial relationships fixed.The suite tests adaptation of motions and behaviors to different goals.Source: E. LIBERO Experiment Details · p. 16
  • The Object suite requires recognizing and manipulating a unique object in each task.The stated task is picking and placing the object.Source: E. LIBERO Experiment Details · p. 16
  • The Long suite evaluates long-horizon planning and performance over extended task durations.Its challenge is maintaining adaptability throughout longer executions.Source: E. LIBERO Experiment Details · p. 16
  • The 90 suite contains 90 diverse short-horizon tasks across five settings.Source: E. LIBERO Experiment Details · p. 16

F. Real Robot Experiments

The real-robot experiments evaluate MDT on multimodal and long-horizon manipulation using teleoperated play data, sparse language annotations, goal images, and text instructions. The experiments also reveal failures on ambiguous object-placement tasks and precise door or tray actions.

Experimental setup

  • The robot setup contains a toy kitchen with 20 diverse tasks and evaluates policies from two static camera perspectives.The environment includes a microwave, oven, cooler, sink, banana, pot, and toaster with toast.Source: F. Real Robot Experiments · p. 16, Figure 7 · p. 18

Data collection

  • Teleoperated play data were collected at 6 Hz with proprioceptive signals and two camera images, using normalized desired joint positions as actions.The data came from a leader-follower robot setup.Source: F. Real Robot Experiments · p. 16
  • Only 360 short-horizon segments received language annotations, with GPT-4 generating varied instructions for each task.The labeled segments were sampled from the uncurated play dataset and described by a human before instruction diversification.Source: F. Real Robot Experiments · p. 17

Training and evaluation

  • Policies were trained for about 24 hours on four GPUs over roughly 100 epochs, selecting checkpoints primarily by validation loss.MT-ACT was instead evaluated using the last epoch because prior benchmark experience suggested later improvement despite rising validation loss.Source: F. Real Robot Experiments · p. 17
  • Evaluation used 10 goal images and 10 text instructions per task, five trials from unseen noisy starts, and four long-horizon chains of five or six tasks.The long-horizon chains included Random, Open Close All, Stovetop Sink, and Oven sequences.Source: F. Real Robot Experiments · p. 17

Results and failure cases

  • Policies often misunderstood similar banana- and pot-placement goals and missed door-closing or oven-tray actions by a few millimeters.The authors hypothesize that sparse labels, similar states, tight actions, varying openness, and quick demonstrations contributed to these failures.Source: F. Real Robot Experiments · p. 17
  • In real-robot goal-image rollouts, 4 out of 6 tasks succeeded, while the robot failed to open the oven door and opened the ice box instead.The result is reported for the illustrated goal-image-conditioned rollouts.Source: Figure 9 · p. 20

Related papers

Loading 2407.05996v1…