PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies

Jingqiu Wang, Yan Wang

arXiv:2609.39324v1cs.ROcs.CVcs.LG

TL;DR

Sparse action supervision and holistic future-scene guidance leave local, timestep-specific visual changes under-characterized. MotionWeave grounds action horizons in current visual tokens and composes adjacent-horizon motion cues; on six MetaWorld tasks, it achieves a 75.3% average success rate, an 8.6% absolute gain over π0.

  • Problem

    Sparse action labels and holistic future-visual guidance leave local interaction changes and timestep-specific correspondence between actions and visual changes under-characterized.

    Source: §1 · p. 1

  • Method

    MotionWeave uses action- and proprioception-conditioned horizon queries to ground interaction regions in current visual tokens, then injects adjacent-horizon differences into action tokens through a gated residual.

    Source: §1 · p. 1

  • Results

    75.3% average success rate on six MetaWorld tasks is an absolute gain of 8.6% over the strongest baseline, π0.

    Source: §3.2 · p. 3, §4 · p. 4

  • Takeaways & Limitations

    Aligning localized motion evidence with action horizons improves visuomotor control, especially on tasks requiring sustained interaction.

    Source: §4 · p. 4

    The policy predicts four-step action chunks from current RGB, language, and proprioception; future RGB observations are available only during training.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have recently incorporated world models to provide richer dynamic supervision beyond sparse action labels. However, explicitly predicting future images or videos may include control-irrelevant appearance, while guidance derived from holistic future visual representations and shared global action features may fail to establish timestep-specific correspondence between actions and local visual changes. To address this issue, we propose MotionWeave, a motion-centric future-dynamics framework for action-chunk prediction with two modules: the Action-Induced Motion Grounder (AIMG) and the Horizon Residual Composer (HRC). Specifically, AIMG conditions on action and proprioceptive representations to construct horizon-specific queries that localize interaction regions associated with each future action timestep from current visual tokens. HRC extracts differences between interaction representations at adjacent horizons, encodes them as temporal motion cues, and injects them into action tokens through a gated residual. During training, robot-arm masks rendered from future frames are used to construct KL-based motion-grounding supervision, while inference uses only the current observation. On six MetaWorld tasks, MotionWeave achieves a 75.3% average success rate, an absolute gain of 8.6% over {\pi}0 (66.7%), especially on sustained-interaction tasks. Our code is available at https://github.com/autu-mn/MotionWeave.

1. INTRODUCTION

Sparse action labels and holistic future-scene guidance leave local visual changes and timestep-specific action correspondence under-specified. MotionWeave pairs horizon-specific interaction grounding with adjacent-horizon motion composition to guide action chunks.

1. INTRODUCTION

  • Sparse action supervision and holistic future-visual guidance leave local interaction changes and timestep-specific action correspondence insufficiently characterized.Source: §1 · p. 1
  • MotionWeave localizes action-relevant dynamic regions and uses their temporal evolution to guide chunk generation instead of reconstructing complete future observations.Source: §1 · p. 1
  • AIMG forms action- and proprioception-conditioned horizon queries over current visual tokens, with future robot-arm masks supervising attention during training.Source: §1 · p. 1, Figure 1 · p. 2
  • HRC differences adjacent-horizon interaction features and injects the resulting motion cues into action representations through a gated residual.Source: §1 · p. 1, Figure 1 · p. 2
  • 75.3% average success rate on six MetaWorld tasks is an absolute gain of 8.6% over π0.Source: §1 · p. 1

2. METHODOLOGY

MotionWeave predicts four-step action chunks from current images, language, and proprioception, using future observations only to supervise motion grounding during training. AIMG localizes horizon-specific visual evidence, and HRC turns changes between those representations into gated residual corrections to action tokens.

2.1. Overall Framework

  • The policy predicts four actions from current RGB, language, and proprioception; future RGB is used only for training-time motion supervision.At inference, the mask-rendering and KL branches are removed.Source: §2.1 · p. 2, §2.4 · p. 3

2.2. Action-Induced Motion Grounder (AIMG)

  • AIMG uses action- and proprioception-conditioned horizon queries to attend to current visual tokens and produce horizon-aligned interaction representations.Training masks rendered from future robot-arm frames provide horizon-specific motion targets, downsampled to the visual-token grid without manual annotation.Source: §2.2 · pp. 2–3

2.3. Horizon Residual Composer (HRC)

  • HRC differences adjacent-horizon interaction representations, then uses motion differences to query action representations for timestep-specific residual corrections.A proprioception-conditioned gate controls residual injection and preserves the original control prior.Source: §2.3 · p. 3

2.4. Training Objective

  • Training combines flow-matching action loss with KL motion-grounding supervision that penalizes attention outside each horizon’s robot-arm mask.The KL term is normalized by the entropy of a uniform distribution over visual-token positions.Source: §2.4 · p. 3

3. EXPERIMENTS

Across six MetaWorld tasks, MotionWeave reaches a 75.3% average success rate, outperforming the strongest baseline while leading on five tasks. Ablations show gains from horizon-wise grounding and temporal composition, with performance sensitive to query count and motion-loss weight.

3.1. Experimental Setup

  • Evaluation uses six MetaWorld tasks, 25 expert trajectories per task, and 25 episodes per task under shared initialization seeds.Each trajectory contains 175 timesteps; the input is a single-view RGB image, language instruction, and proprioceptive state, with four-step action supervision.Source: §3.1 · p. 3

3.2. Main Results

  • 75.3% average success rate gives MotionWeave an 8.6% absolute gain over π0, with the largest advantages on sustained-interaction tasks Disassemble and Shelf Place.MotionWeave leads on five of six tasks; π0 performs better on Pick Place (72.0% vs. 60.0%), where target displacement is short.Source: §3.2 · p. 3, Table 1 · p. 4

3.3. Ablation Studies

  • Adding AIMG without grounding raises average success from 58.0% to 62.0%, grounding raises it to 70.0%, and adding HRC raises it to 75.3%.The ablation separates horizon-specific queries, explicit attention supervision, and adjacent-horizon temporal composition.Source: §3.3 · pp. 3–4, Table 2 · p. 4
  • Four horizon queries and λmotion = 0.05 each yield 75.3% average success, while six or eight queries and larger loss weights produce lower rates.The query-count optimum matches the action chunk length H = 4; redundant queries introduce competing attention.Source: §3.3 · p. 4

4. CONCLUSION

MotionWeave grounds action chunks in localized visual changes and adjacent-horizon motion cues, achieving a 75.3% average success rate on six MetaWorld tasks. The conclusion identifies transfer beyond simulation as future work.

4. CONCLUSION

  • AIMG localizes interaction regions using action- and proprioception-conditioned horizon queries, while HRC injects adjacent-horizon motion cues into corresponding action tokens through a gated residual.Future-frame masks supervise grounding during training; inference uses only the current observation.Source: §4 · p. 4
  • Four horizon queries and λmotion = 0.05 each correspond to a 75.3% average success peak in the reported sensitivity curves.The four-query setting matches action chunk length H = 4.Source: Figure 3 · p. 4
  • 75.3% average success on six MetaWorld tasks is an 8.6% absolute gain over π0, with improved performance on five tasks and particular strength in sustained interaction.The conclusion says transfer to real-robot settings, unseen tasks, and hardware platforms remains for future investigation.Source: §4 · p. 4

Related papers

Loading 2609.39324v1…