PaperlayerView exact revision on arXiv

Source-linked AI summary

FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model

Xiaoxu Xu, Hao Li, Jinhui Ye, Yilun Chen, Jia Zeng, Xinyi Chen, Linning Xu, Dahua Lin, Weixin Li, Jiangmiao Pang

arXiv:2603.10712v1cs.RO

TL;DR

Future guidance in VLAs has struggled to model visual constraints and continuous action dynamics together. FutureVLA uses continuous clips, separate visual and motor streams, and latent alignment to transfer predictive priors. It improves VLA performance across simulated and real-world benchmarks.

  • Problem

    Future guidance methods struggle to capture the interplay between visual geometry and motor actions, with explicit reconstruction favoring appearance and sparse implicit supervision disrupting temporal continuity.

    Source: §1 · p. 2

  • Method

    FutureVLA pretrains on continuous clips with decoupled visual and motor streams, then aligns their joint embeddings with downstream VLA representations.

    Source: §1 · p. 3

  • Results

    FutureVLA significantly improves reasoning and execution over other future-guidance methods across simulated and real-world benchmarks, and surpasses robust π0 by 26.7% in real-world manipulation.

    Source: §1 · p. 4, §5 · p. 14

  • Takeaways & Limitations

    The two-stage approach transfers future-aware priors into diverse VLA policies without altering their inference architectures.

    Source: §5 · p. 14

    For contact-rich operations such as whiteboard erasing, visual constraints alone can be insufficient; the authors identify tactile or force-torque feedback as a potential addition.

    Source: §A.4 · pp. 26–27

Abstract

from arXiv · show

Predictive foresight is important to intelligent embodied agents. Since the motor execution of a robot is intrinsically constrained by its visual perception of environmental geometry, effectively anticipating the future requires capturing this tightly coupled visuomotor interplay. While recent vision-language-action models attempt to incorporate future guidance, they struggle with this joint modeling. Existing explicit methods divert capacity to task-irrelevant visual details, whereas implicit methods relying on sparse frame pairs disrupt temporal continuity. By heavily relying on visual reconstruction, these methods become visually dominated, entangling static scene context with dynamic action intent. We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling. To this end, we propose FutureVLA, featuring a novel Joint Visuomotor Predictive Architecture. FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information, and then jointly encoding generalized physical priors. Specifically, in the pretraining stage, we leverage heterogeneous manipulation datasets and introduce a Joint Visuomotor Gating mechanism to structurally separate visual state preservation from temporal action modeling. It allows the motor stream to focus on continuous physical dynamics while explicitly querying visual tokens for environmental constraints, yielding highly generalizable joint visuomotor embeddings. Subsequently, in the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors without modifying their inference architectures. Extensive experiments demonstrate that FutureVLA consistently improves VLA frameworks.

1 Introduction

Future guidance in VLAs struggles to model visual constraints and continuous action dynamics together. FutureVLA decouples visual and motor information, then aligns joint visuomotor embeddings with downstream VLA models.

1 Introduction

  • Robotic action depends on visual geometry, yet explicit reconstruction diverts capacity to irrelevant details and sparse implicit guidance disrupts temporal continuity.Both approaches also entangle appearance variation with physical state transitions.Source: §1 · pp. 2–3
  • FutureVLA processes continuous multi-frame clips and separates visual state constraints from motor dynamics, conditioning the motor stream on visual information.The resulting embeddings integrate static environmental constraints with continuous control intents.Source: §1 · p. 3
  • Joint Visuomotor Gating decouples visual and motor token objectives, while gated cross-attention lets motor tokens query spatial affordances selectively.Pretraining uses heterogeneous manipulation datasets; post-training aligns downstream VLA representations with future-aware embeddings.Source: §1 · p. 3
  • 11.4% absolute average gains and 9.4% on SimplerEnv over unguided baselines, plus 26.7% over robust π0 in real-world manipulation, are reported.The gains are reported with joint visuomotor embedding guidance.Source: §1 · pp. 3–4

2 Related Work

Related work describes VLA models and two approaches to future guidance. The paper positions structured joint visuomotor prediction as a response to their limitations.

2 Related Work

  • Most VLA models [15] [19] [31] remain largely reactive, encoding temporal dependencies implicitly and limiting long-horizon reasoning and physically consistent action evolution.VLA systems extend pretrained VLMs to generate actions from visual observations, language instructions, and optional state signals.Source: §2 · p. 4
  • Explicit future guidance [11] [37] [39] [41] [49] models future observations or goal states, but full-scene reconstruction introduces visual redundancy.Source: §2 · p. 4
  • Implicit guidance [9] [12] learns latent embeddings from temporally separated frame pairs, but can disrupt continuity and entangle visual changes with action dynamics.These methods typically use inverse and forward dynamics models to infer actions and reconstruct future observations.Source: §2 · p. 4

3 Method

FutureVLA pretrains decoupled visual and motor representations from continuous video, then aligns the resulting joint visuomotor embeddings with downstream VLA representations as future-aware temporal priors.

3 Method

  • FutureVLA uses joint visuomotor pretraining followed by embedding-guided VLA post-training to transfer temporal priors to downstream models.Pretraining uses diverse manipulation datasets; post-training aligns the learned embeddings with downstream VLA representations.Source: §3 · p. 4

3.1 Joint Visuomotor Embedding Pretraining

  • Continuous video clips are encoded into temporal tokens, with visual tokens reconstructing the initial frame and motor tokens supervised by action chunks.A frozen pretrained 3D-VAE produces compact tokens while preserving temporal structure.Source: §3.1 · pp. 5–6, Figure 2 · p. 5
  • Motor tokens model continuous physical dynamics and use gated cross-attention to selectively query spatial constraints from visual tokens.A learnable scalar controls the contribution of cross-attended embeddings to updated action representations.Source: §3.1 · p. 6
  • Visual tokens reconstruct the initial frame using its pretrained 3D-VAE latent embedding as the target.The decoder maps visual context to a reconstructed latent representation, reducing computational overhead relative to image reconstruction.Source: §3.1 · p. 6
  • Pretraining supports OFT-style action regression with MAE loss and GR00T-style action modeling with conditional flow matching.Both action heads use the joint visuomotor embeddings; the GR00T-style formulation conditions on them while modeling action chunks.Source: §3.1 · p. 6

3.2 Joint Visuomotor Embedding Guided VLA Post-training

  • Post-training freezes the pretrained model to provide future-clip embeddings, then aligns adapted VLA intermediate representations to them while training for action prediction.A lightweight Transformer adapter bridges the VLA representations and joint visuomotor embeddings; the objective combines alignment and action losses.Source: §3.2 · p. 7

4 Experiments

Experiments evaluate FutureVLA on simulation benchmarks and four Franka manipulation tasks, then ablate its guidance, gating, temporal sampling, and horizon. FutureVLA improves over baselines, while the ablations support continuous temporal modeling and visually conditioned motor representations.

4.1 Experimental Setups

  • Evaluations span LIBERO, SimplerEnv on Google and WidowX robots, and four Franka tasks covering grasping, tool use, insertion, and contact-rich control.The study compares original VLA baselines with explicit and implicit future-guidance methods.Source: §4.1 · pp. 7–8

4.2 Main Results

  • On Google-robot SimplerEnv Visual Matching, FutureVLA-GT gains 44.9% over GR00T-N1.5 and FutureVLA-OT gains 30.1% over OpenVLA-OFT; real-world FutureVLA-GT reaches 70.0% average success, 26.7% above π0.The paper also reports consistent improvements across LIBERO tasks, with the largest improvement on Long, and similar trends on WidowX.Source: Table 1 · p. 8, Table 2 · p. 8, §4.2 · p. 9

4.3 Ablation Studies

  • On WidowX, JVPM alignment improves average performance by 9.4% for both GR00T-style and OFT-style models versus unguided counterparts.The ablation isolates latent-embedding alignment during post-training.Source: §4.3 · pp. 9–10, Table 4 · p. 11
  • Decoupling visual and motor tokens without interaction improves performance by 7.2% over Variant (b), which directly processes multi-frame video tokens through Transformer layers to produce motor tokens.Source: §4.3 · p. 11
  • Performance degrades consistently as temporal sampling becomes sparser, supporting continuous multi-frame observations over sparse frame pairs.The tested sampling sizes were 2, 5, 9, and 17 frames; the two-frame input was padded to five frames for the 3D-VAE.Source: §4.3 · p. 12, Figure 4 · p. 12
  • WidowX performance improves as the horizon grows from 9 to 17 frames but declines at 21, so the study selects 17 frames to align visual and action horizons.The action chunking horizon is fixed at 16; shorter inputs provide insufficient coverage, while longer sequences add redundant or misaligned context.Source: Figure 5 · p. 12, §4.3 · p. 13
  • Under a unified architecture, FutureVLA achieves the best results against explicit WorldVLA-style and implicit LAPA-style and Villa-X-style guidance.The comparison controls for backbone differences between guidance methods.Source: Figure 6 · p. 12, §4.3 · p. 14
  • FutureVLA’s latent similarity has the strongest correlation with physical action consistency among the compared methods.Qualitatively, clips with nearby embeddings show consistent robot behaviors; the quantitative comparison uses nearest neighbors and action-consistency scores.Source: Figure 7 · p. 13, Figure 8 · p. 13, §4.3 · p. 14

5 Conclusion

FutureVLA addresses visually dominated embedding entanglement and temporal discontinuity through structurally decoupled visual constraints and continuous action modeling. Its two-stage training transfers future-aware priors to diverse VLA policies, and experiments report improved reasoning and execution across simulated and real-world benchmarks.

5 Conclusion

  • FutureVLA structurally separates static visual constraints from continuous temporal action modeling to extract physically grounded joint visuomotor embeddings.The paper frames this as a response to visually dominated embedding entanglement and temporal discontinuity in future-guided VLAs.Source: §5 · p. 14
  • A pretraining and latent-alignment post-training paradigm transfers future-aware priors to diverse VLA policies without changing their inference architectures.Source: §5 · p. 14
  • Experiments across simulated and real-world benchmarks demonstrate improved reasoning and execution capabilities over other future-guidance methods.The paper presents this as a scalable path toward physically consistent embodied foundation models.Source: §5 · p. 14

A Implementation Details for FutureVLA

The appendix describes FutureVLA’s architecture and training configuration and reports additional experiments intended to validate the method.

A Implementation Details for FutureVLA

A.1 Architecture Overview

FutureVLA encodes 17-frame clips into temporal tokens, divides them into visual and motor streams, and conditions motor tokens on visual tokens through gated cross-attention. A separate visual reconstruction branch pools visual tokens to predict the first-frame latent embedding.

A.1 Architecture Overview

  • The pretraining model encodes 17 consecutive 224 × 224 RGB frames with a frozen pretrained 3D-VAE, producing video tokens at 1960 × 48 resolution.The encoder requires input lengths of 4N + 1, and temporal block encoding produces the compact representation.Source: §A.1 · p. 19
  • Two Transformer layers process the video tokens before they are divided into 980 visual tokens and 980 motor tokens.The visual stream then uses a three-layer Transformer, while the motor stream uses one Transformer layer.Source: §A.1 · p. 19
  • Motor tokens query visual tokens as keys and values through cross-attention, with a learnable gated update added to the motor representation.This visually conditioned update is applied iteratively three times.Source: §A.1 · p. 19
  • The visual branch reconstructs the first-frame 3D-VAE latent embedding, which has 392 × 48 resolution.A QueryPool module uses learnable queries and softmax attention weights to aggregate visual tokens to the target token count.Source: §A.1 · p. 19
  • Pooled visual representations pass through a three-layer Transformer decoder to predict the reconstructed latent embedding.Source: §A.1 · p. 19

A.2 Training Details

The appendix documents a heterogeneous pretraining mixture and the optimization and post-training protocols used to adapt FutureVLA to downstream tasks.

A.2 Training Details

  • Pretraining draws primarily on OXE and LIBERO, totaling 15.6 million frames, with dataset composition shown in Fig. 9.Single-view datasets duplicate the same image to create a two-view input; the pretraining action chunk size is 16.Source: §A.2 · pp. 19–20, Figure 9 · p. 20
  • Pretraining uses batch size 256, learning rate 1 × 10−5, and a 5,000-step warm-up, taking about three days on 16 NVIDIA A100 GPUs.Source: §A.2 · p. 20
  • Post-training uses Qwen3-VL-4B-Instruct initialized from pretrained weights, with a 5,000-step warm-up and cosine decay of the latent-alignment coefficient β.GR00T-style and OFT-style paradigms are implemented according to their respective VLA-module/action-head interactions.Source: §A.2 · p. 20

A.3 Ablation Study Details

The ablations report gains from joint visuomotor guidance across simulated and real-world evaluations, and examine robustness, reconstruction choices, video encoding, and action-consistency evaluation.

LIBERO-Plus evaluation

  • On LIBERO-Plus, FutureVLA-GT gains 10.1% in average overall success rate over OpenVLA-OFT, and improves success rate by 54.7% over WorldVLA and 35.8% over UniVLA.Source: §A.3 · p. 20, Table 6 · p. 21

Google robot JVPM ablation

  • On the Google robot, JVPM raises average success rate by 6.9% and 4.4% for GR00T-style Visual Matching and Variant Aggregation, and by 15.8% and 18.8% for OFT-style.Source: Table 7 · p. 21, §A.3 · p. 21

LIBERO JVPM ablation

  • On LIBERO, JVPM improves average success rate by 1.9% for GR00T-style and 1.6% for OFT-style, including a 4% GR00T-style gain on Long tasks.Source: Table 8 · p. 22, §A.3 · p. 22

Real-world JVPM ablation

  • Across four real-world contact-rich tasks, JVPM raises average success rate by 21.7% for GR00T-style and 18.4% for OFT-style models.In whiteboard erasing, the reported gains are 40.0% for FutureVLA-GT and 20.0% for FutureVLA-OT.Source: Table 10 · p. 22, §A.3 · p. 23

Temporal encoding ablation

  • Multi-frame 3D-VAE encoding achieves higher task success rates and lower per-step training time than independent single-frame VQ-GAN encoding.The authors attribute the result to explicit capture of inter-frame motion dynamics and more coherent temporal representations.Source: Table 12 · p. 23, §A.3 · p. 23

Reconstruction ablations

  • Reconstructing the first rather than last frame improves average performance by 3.1% across metrics; reconstruction-loss weights that are too large or small also degrade performance.The first frame serves as a static geometric anchor, whereas last-frame reconstruction asks visual tokens to represent future visual dynamics.Source: §A.3 · pp. 23–24, Table 11 · p. 23, Table 13 · p. 24

Robustness to visual perturbations

  • Under task-irrelevant visual perturbations, FutureVLA has lower embedding deviation and more robust action prediction than visually entangled LAPA and Villa-X.Fig. 10 illustrates the perturbations, while Table 14 reports embedding and action-error measures.Source: §A.3 · p. 24, Figure 10 · p. 24, Table 14 · p. 25

PAAC and temporal evaluation

  • PAAC uses DTW to account for temporal misalignment while retaining sensitivity to action magnitude, then maps distance to a normalized similarity score.Temporal sparsity is tested with 2, 5, 9, and 17 sampled frames; the two-frame configuration is padded to five for 3D-VAE compatibility.Source: §A.3 · pp. 25–26

A.4 Limitations and Future Work

The authors identify a limitation of relying only on visual constraints for contact-rich operations and propose adding tactile or force-torque feedback.

A.4 Limitations and Future Work

  • For contact-rich operations such as whiteboard erasing, visual constraints alone can be insufficient; future versions could incorporate tactile or force-torque signals.Source: §A.4 · pp. 26–27

A.5 Simulation Evaluation Details

The appendix specifies the simulation and real-world evaluation settings, then describes the VLA and future-guidance baselines used for comparison.

A.5 Simulation Evaluation Details

  • SimplerEnv tests Google-robot spatial generalization and perturbation robustness, plus WidowX Visual Matching; LIBERO covers four 10-task suites, and LIBERO-Plus tests seven perturbation dimensions across 10,030 tasks.Google tasks include four representative manipulations; WidowX evaluation follows Bridge data distribution, and SimplerEnv uses CogAct protocols.Source: §A.5 · p. 27

A.6 Real-World Robot Evaluation Details

  • Real-world evaluation uses a Franka Research 3 with a 7-DoF manipulator, RGB-D cameras, and four tasks spanning assembly, insertion, tool use, and contact-rich wiping.The setup collects 300 post-training trajectories, 75 per task, recorded at 5 Hz.Source: §A.6 · p. 28

A.7 Details of Compared Models

  • Compared original VLA baselines include OpenVLA-OFT, π0, and GR00T-N1.5, alongside original multimodal-observation-to-action baselines.π0 uses flow matching for smooth, high-frequency trajectories and is trained across seven platforms and 68 tasks; GR00T-N1.5 maps multimodal observations to continuous actions.Source: §A.7 · p. 29
  • Explicit future-guidance comparisons include WorldVLA, which predicts future observations, and DreamVLA, which forecasts structured future representations for action planning.Source: §A.7 · p. 29
  • Implicit-guidance comparisons include LAPA, UniVLA, and Villa-X, which model latent actions from video or heterogeneous data and, for Villa-X, robot-action distributions.Source: §A.7 · pp. 29–30
Loading 2603.10712v1…