PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality-Autoregressive World-Action Models

Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski

arXiv:2609.17524v1cs.RO

TL;DR

WAMs must determine how to combine future visual modalities beyond RGB to support action prediction. ModAR autoregressively denoises multiple modalities before actions and evaluates formulations, modality sets, and data mixtures from scratch. It achieves the strongest reported performance across evaluated formulations and data scales, while remaining compute-efficient, but the study is limited in task and instruction diversity and incurs higher inference latency.

  • Problem

    The best way to combine future RGB, depth, pretrained features, and point tracks within world-action models remains an open question.

  • Method

    ModAR autoregressively denoises multiple future modalities in sequence, conditioning each prediction on earlier modalities before denoising actions.

  • Results

    ModAR achieves the highest average success rate at every evaluated data scale; point tracks, DINO features, and depth help, while additional future RGB provides no consistent benefit.

  • Takeaways & Limitations

    Modality-autoregressive prediction is a promising alternative to relying solely on future RGB prediction for world-action modeling.

  • Takeaways & Limitations

    The experiments cover few tasks, use discrete task labels rather than language instructions, and sequential generation increases inference latency.

Abstract

from arXiv · show

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-$π$ on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately $20\times$ fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.

I. INTRODUCTION

World-action models combine future-observation prediction with action prediction, but the best way to use multiple future representations remains unresolved. ModAR addresses this by autoregressively generating modalities before actions and systematically comparing formulations, targets, and data sources.

  • I. INTRODUCTION: WAMs can learn future observations from actionless data, enabling broader supervision than action-labeled robot demonstrations alone.Future predictions can support action learning during training or condition action generation at deployment.
  • I. INTRODUCTION: ModAR autoregressively denoises multiple future-observation modalities before generating actions.Each modality can condition on earlier predictions, with actions generated last.
  • I. INTRODUCTION: Point tracks, DINO features, and depth provide additive gains, whereas additionally predicting future RGB offers no consistent benefit.These representations emphasize motion, semantic, geometric, or physical structure beyond visual appearance.
  • I. INTRODUCTION: ModAR’s sequential formulation differs from unified, disjoint, and action-only WAMs in how future predictions interact with action prediction.Unified models denoise future and action streams together, while disjoint models separate them and omit future prediction at inference.
  • I. INTRODUCTION: The study trains models from scratch to isolate the effects of WAM formulation, predicted representations, and actionless-data scale.This controlled setup avoids confounding these factors with pretrained video-model initialization.

B. Multimodal generation

Multiple future representations can provide complementary supervision, and their generation order can determine how effectively they help one another. ModAR extends this idea by generating structured modalities before increasingly detailed ones.

  • B. Multimodal generation: Jointly predicting multiple future representations can enrich shared features through complementary inductive biases.Co-training objectives can improve task performance relative to single-task training.
  • B. Multimodal generation: The evaluated formulations differ in whether future observations and actions are generated sequentially, jointly, independently, or not at all.ModAR uses sequential generation; Unified jointly generates streams, Disjoint separates them, and Action-only predicts actions directly.
  • B. Multimodal generation: ModAR generates future modalities autoregressively, using earlier predictions as structured context for later targets.Its order proceeds from point tracks and DINO features to depth and RGB.

III. METHOD

ModAR autoregressively denoises future modalities one at a time, using completed predictions as context before generating actions. It conditions on observations, robot configuration, task identity, and previously generated modalities, while training supports both action-labeled and actionless examples.

  • Problem formulation: The model predicts future RGB, depth, DINO features, and point tracks from the current multimodal observation, robot configuration, and task embedding.The prediction horizon contains J future targets per modality, while actions are predicted at every control step through the horizon.
  • Modality-autoregressive generation: ModAR generates future modalities sequentially, using each completed prediction as context and conditioning the final action chunk on all generated modalities.The generation order is tracks → DINO → depth → RGB, followed by actions.
  • Block-causal generation: A block-causal mask prevents target leakage by allowing each modality block to attend only to the observation and previously completed modality blocks.Training uses clean context copies and noisy prediction copies for every target modality.
  • Training robustness: Context noise during training exposes later predictions to imperfect earlier generations, mitigating cascading errors during modality-autoregressive inference.Noise is added independently to context blocks during training and omitted during inference.
  • Training objective: All output streams use a JiT-style x-prediction objective, with the same objective applied to action chunks and action loss included only for action-labeled examples.The total loss combines supervised modality terms with the action term when labels are available.
  • Inference: Inference initializes each modality and the action stream from Gaussian noise, denoises them sequentially, and caches prior keys and values to avoid repeated computation.Each clean prediction is re-embedded as context for the next modality.

IV. EXPERIMENTS

The experiments compare WAM formulations and predicted-modality subsets using success-rate tables and simulation evaluations. The supplied experiment visuals report per-task success rates across formulations and modality subsets.

  • Evaluation scope: The experiment section evaluates WAM formulations, actionless-data scaling, predicted modality sets, comparison with Flex-π, and real-world bimanual manipulation.These evaluations combine simulation and real-world settings.
  • Modality comparison: Table II reports per-task success rates for predicted-modality subsets comprising RGB, depth, point tracks, and DINO features.The comparison is made at total-demonstration scale D.
  • Formulation comparison: Table I reports per-task success rates for Action-only, Independent-noise, Disjoint, Unified, and ModAR formulations.The table is organized by total demonstration scale D.

A. Experimental setup

The study evaluates controlled WAM comparisons on six RoboTwin simulation tasks and three real-world bimanual tasks. It varies total demonstration counts and data mixtures while holding key training components consistent across baselines.

  • Simulation setup: Simulation uses six RoboTwin tasks with 50, 250, or 1,250 total demonstrations, retaining 50 action-labeled demonstrations and adding actionless demonstrations.Each model is multitask-trained across all six tasks and evaluated on 50 held-out initial conditions per task.
  • Real-world setup: Real-world evaluation covers cup stacking, towel folding, and drawer placement with bimanual YAM arms.The study collects teleoperated robot demonstrations, in-domain actionless human demonstrations, and EgoDex demonstrations.
  • Real-world evaluation: Each real-world method and data mixture is evaluated with 30 rollouts per task using varied initial object poses.One multitask model is trained for all three real-world tasks.
  • Baselines: Controlled baselines share the same backbone, action-labeled data, and optimization budget while differing in how future observations inform action generation.The baselines include Unified, Disjoint, Independent-noise, Flex-π, and Action-only.

B. Simulation results

Across simulation studies, ModAR's autoregressive formulation achieves the strongest performance across data scales, benefits from actionless data and structured modalities, and remains superior under matched sampling or action-prediction evaluation.

  • Formulation comparison: ModAR achieves the highest average success rate at every evaluated actionless-data scale.The authors hypothesize that early modalities provide structured scratchpad context for later, more detailed targets.
  • Scaling with actionless data: Adding 1,200 actionless demonstrations raises ModAR's average success rate from 66% to 76%, compared with a 1% improvement for Unified.Disjoint generation outperforms Action-only on average, but its performance decreases as actionless data increases.
  • Separate inverse-dynamics model: Using the same separately trained inverse-dynamics model, ModAR still outperforms Unified, indicating that ModAR's predicted futures are more useful for action prediction.This evaluation replaces both models' native action predictors to remove the asymmetry in how their action predictions use generated futures.
  • Modality comparison: Predicting point tracks, DINO features, and depth generally adds or preserves performance, whereas adding future RGB provides no consistent gain.The best modality set varies by task, and the authors hypothesize that RGB adds high-variance appearance details with limited information beyond structured targets.
  • Sampling-step comparison: With 40 matched Euler steps, ModAR remains at 75% average success while Unified falls from 67% to 59%, Disjoint from 55% to 54%, and Independent-noise rises from 34% to 37%.The additional sampling budget therefore does not close the performance gap.
  • Inference latency: End-to-end ModAR inference takes 147.9 ms, or 6.76 Hz, when generating all four future-observation modalities and actions.This latency is reported on a single NVIDIA GeForce RTX 5090 GPU.

C. Ablations

Ablations show that context noise and the structured modality order are important to ModAR's performance, while future RGB contributes least among the evaluated modalities.

  • Context noise: Removing context noise reduces average success from the full model's 75% to 63%, showing its importance for preventing compounding autoregressive errors.The ablation is conducted at the 250-demonstration scale.
  • Modality order: Reversing the modality order to RGB → depth → DINO → tracks lowers average success to 65%, supporting structured modalities as scratchpads for detailed ones.A full systematic comparison of modality orderings remains future work.
  • Ablating modalities: Removing tracks, DINO, or depth lowers average success to 61%, 65%, or 70%, respectively, while removing future RGB leaves performance unchanged.Thus, RGB contributes least among the four future-observation modalities in this setting.

D. Real-world experiments

On three real-world bimanual tasks, ModAR outperforms the compared baselines and improves as actionless human demonstrations are added, including out-of-domain video data.

  • Modality selection: Because RGB gave no consistent simulation benefit while increasing costs, the real-world WAMs predict tracks, DINO, and depth only.This design choice excludes future RGB from the real-world models.

E. Implementation details

The implementation uses a shared diffusion-transformer backbone with modality-specific experts, autoregressive flow-matching details, and paired robot or actionless human data.

  • Inputs and outputs: Models use a single-camera 168 × 224 observation, predict targets at t + 8 and t + 16, and output a dense 16-step action chunk with 14-dimensional dual-arm configurations.Policies replan every 16 steps; RGB, depth, and track queries use a 12 × 16 grid of 14 × 14 patches.
  • Architecture: The shared DiT has six width-384 transformer blocks, followed by modality-specific experts for DINO, depth, RGB, tracks, and actions.The shared blocks fuse cross-modal information before within-modality processing.
  • Optimization: Training uses AdamW with learning rate 10^-4, global batch size 48, bfloat16, EMA decay 0.999, and 1.2M optimizer steps.Action-labeled and actionless pools contribute equal-sized batches and equally weighted losses.
  • Flow and sampling: ModAR uses maximum context noise β = 0.5, unit loss weights, modality-specific logit-normal flow times, and eight Euler steps per generated stream.Unified uses one shared flow timestep, whereas Independent-noise samples each stream's timestep independently.
  • Real-world data collection: Real-world robot demonstrations are collected by teleoperation with paired teacher arms, while a fixed third-person ZED stereo camera supplies RGB and depth for robot and actionless human demonstrations.The same camera setup supports both robot and human data collection.

V. CONCLUSION

ModAR autoregressively denoises multiple future-observation modalities before predicting actions. It achieves the strongest reported performance across controlled simulation and three real-world bimanual tasks, with complementary gains from point tracks, DINO features, and depth but no consistent benefit from future RGB.

  • Conclusion: ModAR achieves the highest average success rate among representative WAM formulations in simulation and the highest success rate among baselines on three real-world bimanual tasks.Its performance also improves as actionless data and human video demonstrations are added.
  • Conclusion: Point tracks, DINO features, and depth provide complementary gains, whereas additionally predicting future RGB provides no consistent benefit in the experiments.
  • Conclusion: Modality-autoregressive prediction is presented as a promising alternative to relying solely on future RGB prediction for world-action modeling.

VI. LIMITATIONS

The evaluation scope is limited: experiments cover few tasks and use discrete task labels rather than language instructions. Consequently, broad generalization across tasks, objects, or scenes is not established, and sequential generation increases inference latency.

  • Limitations: Experiments cover a limited number of tasks and use discrete task labels rather than language instructions, so broad generalization is not established.The unestablished scope includes generalization across tasks, objects, and scenes.
  • Limitations: Sequentially generating multiple modalities increases inference latency relative to simultaneous or action-only generation.
Loading 2609.17524v1…