PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Action or World-Action? Just Action Models

Yonghao Yu, Lang Huang, Runyi Li, Toshihiko Yamasaki

arXiv:2610.37219v1cs.CVcs.RO

TL;DR

VLA and WAM comparisons mix backbone choice, future-observation supervision, and inference-time machinery, leaving their individual contributions unclear. JAM separates these choices by training an action policy with frozen future-representation targets, then evaluates the recipe across fixed architectures and backbone families. It improves matched FastWAM and ImageWAM results on LIBERO and LIBERO-Plus, with similar or slightly higher RoboTwin success.

  • Problem

    VLA–WAM success-rate comparisons alone cannot identify whether differences come from backbone initialization, future-observation objectives, or inference machinery.

    Source: §1 · p. 1

  • Method

    JAM trains a pretrained visual backbone and action expert with action and foresight losses, using a training-only head to predict frozen encodings of recorded futures.

    Source: §1 · p. 2

  • Results

    JAM improves matched policies on LIBERO and LIBERO-Plus and achieves similar or slightly higher RoboTwin success; LIBERO-Plus FastWAM rises from 55.31% to 64.54%.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Strong foresight performance is attainable with generative, self-supervised, or vision-language backbones, although cross-backbone comparisons do not isolate pretraining effects.

    Source: §1 · p. 2, §5 · p. 10

    The study evaluates simulation-trained policies, and unsafe deployment without physical validation and failures under distribution shift remain relevant risks.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

The vision-language-action (VLA) and world-action model (WAM) communities debate whether control policies should inherit pretrained knowledge from vision-language models or from visual generators. The debate, however, bundles three separable choices: backbone initialization, the objective applied to future observations, and whether the machinery for that future loss is kept at inference. A gap between a VLA and a WAM therefore cannot simply be attributed to any one of them. We show that decoupling these choices yields a common framework for both families, which we term Just Action Model (JAM): a pretrained visual backbone with an action expert, trained with an action loss and a foresight loss. The foresight loss trains current-observation features to predict frozen encodings of the recorded future through a prediction head discarded after training, so no visual generation is involved in either training or inference, and the backbone can be chosen freely. We first compare objectives within fixed policy architectures, then evaluate the foresight recipe across backbone families. The resulting JAM policies outperform their matched FastWAM and ImageWAM counterparts on LIBERO and LIBERO-Plus, and achieve similar or slightly higher success rates on RoboTwin. Compared with FastWAM, JAM improves the success rates from 97.0%/55.3%/91.2% to 98.2%/64.5%/91.4%, respectively. With cached foresight targets, JAM reduces estimated online training FLOPs per sample per update by 73% relative to FastWAM. Our results show that future supervision improves control without visual generation or additional inference machinery, and strong foresight performance is attainable with either generator-initialized or vision-language backbones. Project Page: https://y0uroy.github.io/JAM

1 INTRODUCTION

VLA–WAM comparisons bundle backbone initialization, future-observation objectives, and inference machinery, so success-rate differences alone cannot identify which choice matters. JAM separates these choices and uses future representations as training-only supervision; fixed-architecture and cross-backbone evaluations show the recipe’s control performance and scope.

1 INTRODUCTION

  • LIBERO-Plus success rises from 55.31% to 64.54% for FastWAM and from 65.27% to 79.95% for ImageWAM when their generative objectives are replaced with foresight.The comparisons keep each policy’s architecture, data, optimizer, training budget, and deployment path fixed.Source: §1 · p. 2
  • With cached targets, JAM reduces estimated online training FLOPs per sample per update by 73.0% for FastWAM and 36.8% for ImageWAM.These estimates compare the foresight recipe against the corresponding generative-objective policies.Source: §1 · p. 2
  • On LIBERO-Plus, JAM reaches 79.95% with Ovis-U1, 70.39% with DINOv2-L, and 81.14% with Qwen3.5-2B, versus 65.27% for ImageWAM (Ovis-U1).The foresight recipe is evaluated with generative, self-supervised, and vision-language backbones.Source: §1 · p. 2
  • JAM uses a pretrained visual backbone and action expert, training with action and foresight losses while discarding the future-target branch after training.A prediction head maps current-observation features to frozen future-anchor targets; recorded futures never enter the action path.Source: §1 · p. 2
  • Future-offset targets outperform current-offset targets, and the best target representation depends on the backbone.A plain VAE suffices for the Wan video backbone; the strongest JAM (Ovis-U1) result uses targets from the pretrained Yak backbone.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work combines pretrained visual models with action prediction or predictive supervision in several ways. JAM’s related-work distinction is where prediction is applied and how its future target space is selected.

2 RELATED WORK

  • FastWAM retains video co-training without test-time video generation [Yuan et al., 2026], while ImageWAM uses an image-editing backbone and future-image target.JAM replaces their visual-generation objectives while preserving each policy’s architecture and deployment path.Source: §2 · p. 3
  • Joint-embedding methods avoid pixel reconstruction [Assran et al., 2025; Daithankar et al., 2026], while Mirai and Video-Mirai apply foresight to image and video generation [Yu et al., 2026b;a].Var-JEPA studies a variational formulation on tabular data [G¨ogl & Yau, 2026].Source: §2 · p. 3
  • FLARE mixes future, state, and action tokens through self-attention in its action DiT, while JEPA-WAM supervises a shared action-generating predictor with dense joint current–future targets in frozen V-JEPA space (Lin et al., 2026).Source: §2 · p. 3
  • JAM differs from FLARE by applying prediction through a separate training-only head on existing visual-backbone tokens, preserving matched action computation.Unlike JEPA-WAM’s joint current–future V-JEPA target [Lin et al., 2026], JAM supervises future-position features in a target space chosen for each backbone.Source: §2 · p. 3

3 JAM: JUST ACTION MODELS

JAM pairs a pretrained visual backbone with an action expert and uses recorded futures as targets for a training-only foresight branch. The recipe spans generative, self-supervised, and vision-language backbones, while deployment retains only the backbone and action expert.

3.1 PROBLEM FORMULATION

  • JAM combines an action loss with foresight supervision from a frozen future anchor, replacing visual generation while leaving the deployed policy architecture unchanged.Recorded futures supervise current-observation features but never enter the action path.Source: §3.1 · p. 4

3.2 FORESIGHT SUPERVISION

  • A frozen future anchor encodes recorded futures at selected temporal offsets; a trainable head predicts those features from policy tokens, and cosine distance is averaged over tokens and offsets.With fixed preprocessing and conditioning, target features can be cached.Source: §3.2 · p. 4
  • At inference, JAM discards the anchor branch and prediction head, encodes the current observation and instruction once, and reuses that context across action flow-matching steps.This matches FastWAM and ImageWAM deployment, while avoiding visual generation during training as well.Source: §3.2 · pp. 4–5

3.3 INSTANTIATIONS ACROSS PRETRAINING FAMILIES

  • For generative backbones, JAM uses only the clean current frame and preserves the policy architecture and action-expert initialization of the matched FastWAM and ImageWAM setups.Additional targets isolate pretrained generator features from random-initialized features and VAE-tokenizer features.Source: §3.3 · p. 5
  • The DINOv2 instantiation uses an image encoder without language input or a generative head, with a separate text encoder supplying instruction features to the action expert.The text encoder does not provide input to the visual backbone.Source: §3.3 · p. 5
  • The Qwen3.5-2B instantiation uses a vision-language backbone whose tokens are mixed causally with image, text, and action tokens by the action expert.This is a VLA trained with an additional foresight loss.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments first compare generation and foresight under matched policy setups, then test foresight across backbone families and benchmark performance, including training cost.

4.2 REPLACING GENERATION WITH FORESIGHT

  • With fixed policy setups, foresight raises ImageWAM from 92.90% to 96.75% on LIBERO and from 65.27% to 79.95% on LIBERO-Plus.For FastWAM, foresight changes LIBERO success from 97.00% to 97.45% and LIBERO-Plus from 55.31% to 55.58%. Future-only offsets [1,2] raise FastWAM LIBERO-Plus success from 44.62% at [0] to 55.58%; adding [0] back yields 54.18%.Source: §4.2 · p. 6, Table 1 · p. 6
  • On LIBERO-Plus, JAM (Wan) performs best with VAE-tokenizer targets at 64.54%, while JAM (Ovis-U1) performs best with pretrained Yak targets at 79.95%.A random Wan target reaches 58.63%, and a random Yak target reaches 72.51%; the preferred target differs by architecture.Source: §4.2 · pp. 6–7
  • On the displayed LIBERO-Long example, JAM with offset [1] makes future-frame changes more readily decodable than ImageWAM or JAM with offset [0].The probe study also reports the highest mean future-frame PSNR for JAM [1] among the compared policies.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 VARYING THE BACKBONE

  • With a pretrained visual backbone, JAM (Wan) reaches 97.45% on LIBERO with pretrained targets and 97.65% with random targets; random-backbone results are 71.15% and 84.40%, respectively.The passage attributes strong control performance to policy-backbone pretraining while noting that a frozen random future anchor remains effective.Source: §4.3 · p. 7, Table 2 · p. 7
  • JAM applies a common action-plus-foresight objective across generative, self-supervised, and vision-language backbones; Ovis-U1, DINOv2-L, and Qwen3.5-2B score 79.95%, 70.39%, and 81.14% on LIBERO-Plus.These four JAM backbone variants exceed ImageWAM (Ovis-U1) on both LIBERO and LIBERO-Plus, though architecture and action-interface differences prevent isolating pretraining effects.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Table 6 · p. 9
  • The attention analysis shows different input allocations: DINOv2-L attends most to image keys, JAM (Ovis-U1) to text keys, and Qwen3.5-2B to action keys.For the example observation, ImageWAM focuses on the mug in the gripper, while JAM on the same backbone places more attention on the second mug named in the instruction.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 BENCHMARK COMPARISONS

  • JAM matches or exceeds generative baselines across RoboTwin 2.0, LIBERO, and LIBERO-Plus; on LIBERO-Plus it has the best average in all three groups.JAM (FLUX.2) has the highest listed RoboTwin 2.0 score at 93.1%; on LIBERO-Plus, JAM exceeds FastWAM and ImageWAM (Ovis-U1) by 9.2 and 14.7 points.Source: Table 4 · p. 8, Table 5 · p. 8, §4.4 · p. 9

4.5 TRAINING COST

  • With cached foresight targets, JAM (Wan) reduces estimated online cost from 15.52 TFLOPs for FastWAM to 4.19 TFLOPs per sample per update, a 73.0% reduction.The corresponding ImageWAM (Ovis-U1) comparison falls from 5.81 to 3.67 TFLOPs, a 36.8% reduction.Source: §4.5 · p. 9

5 CONCLUSION

JAM separates backbone initialization, future-observation supervision, and inference modules, using action and foresight losses while discarding the foresight branch after training. The conclusion reports gains over matched generative baselines, lower cached-target training cost, and applicability across backbone families.

5 CONCLUSION

  • JAM replaces visual generation with foresight and outperforms matched generative baselines on LIBERO and LIBERO-Plus, with similar or slightly higher RoboTwin 2.0 success.The conclusion reports the largest gains on LIBERO-Plus.Source: §5 · pp. 9–10
  • With cached targets, JAM reduces online FLOPs per sample per update by 73.0% versus FastWAM and 36.8% versus ImageWAM.The comparison concerns online training cost, not deployment cost.Source: §5 · p. 10
  • The foresight recipe applies to generative, self-supervised, and vision-language backbones, and the best generator-initialized and vision-language policies achieve comparable performance.The cross-backbone comparison does not isolate the effect of pretraining alone.Source: §5 · p. 10
  • The evaluation covers simulated robot manipulation with public models, code, and datasets; distribution-shift failures and unsafe deployment without physical validation remain relevant risks.The study uses LIBERO, LIBERO-Plus, and RoboTwin 2.0 and involves no human participants or personal data.Source: ETHICS STATEMENT · p. 11
  • The authors make code and selected model weights publicly available.The implementation builds on public FastWAM and ImageWAM code and public benchmark datasets.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 TRAINING SETUP

The training setup varies by backbone recipe and benchmark, while future observations enter only the future-anchor branch. Across backbones, a training-only prediction head aligns current features with frozen future targets alongside the action loss.

A.1 TRAINING SETUP

  • For LIBERO and LIBERO-Plus, the Wan recipe uses 32-step action chunks and 20k updates, while the ImageWAM recipe uses 16-step chunks and 21.7k updates; both use two 224×224 views and batch size 128.RoboTwin uses three camera views and five epochs, with recipe-specific image resolutions, batch sizes, and update counts.Source: §A.1 · p. 15

A.2 BACKBONES AND TARGET INTERFACES

  • Wan targets use selected temporal VAE positions or lifted VAE-direct features, while Ovis-U1 and FLUX.2 use frozen future anchors with current and future frames; Qwen3.5 and DINOv2 use frozen encoders for future-view feature targets.Future observations enter only the future-anchor branch, and target tokens are excluded from the trainable stream for Ovis-U1.Source: §A.2 · pp. 15–16

A.3 PREDICTION HEAD AND LOSS

  • The prediction head maps current features to target-offset representations, and the objective combines action loss with cosine-distance foresight loss over valid examples, tokens, and offsets.Targets receive no gradients, and the anchor branch and head are removed after training.Source: §A.3 · p. 16

B TRAINING-COST ACCOUNTING

The cost accounting distinguishes training-only additions from deployed parameters and estimates online FLOPs under stated approximations. GPU-time profiling and cached-target costs have additional measurement conditions.

B.1 PARAMETERS

  • Deployed parameters match the generative baselines, while training-only target costs differ: Wan reuses its frozen VAE, whereas Ovis-U1 loads a frozen 1.13B Yak18 for online targets.The Wan VAE-direct targets require no additional target parameters.Source: §B.1 · pp. 16–17, Table 9 · p. 17

B.2 FLOPS

  • The FLOP estimate uses 2NT per forward pass, assigns trainable blocks three forward-equivalents or four with ImageWAM checkpointing, and counts frozen anchors as forward-only.Attention-score terms, text encoding, and optimizer updates are omitted.Source: §B.2 · p. 17
  • Wan’s cached VAE-direct targets require encoding only the current frame, while online targets encode the full window; cached-cost estimates exclude one-time extraction and require fixed anchors, preprocessing, temporal indices, and conditioning.GPU-time profiles use one GH200 at batch 16, and Wan’s measured reductions are smaller than its FLOP estimates.Source: §B.2 · p. 17

C ABLATIONS FOR FORESIGHT SUPERVISION

The ablations examine future-anchor choice, supervision depth, loss weight, prediction-head capacity, action-only controls, and whether action prediction requires a separate expert. Results vary across these design choices and backbone-specific perturbations.

C.1 EMA FUTURE ANCHOR

C.2 SUPERVISED DEPTH

C.3 LOSS WEIGHT

  • λF = 0.6 gives the highest average and is the only tested loss weight to exceed FastWAM; Long success ranges from 90.0% to 97.0%.Each readout offset receives weight λF/|K|, so adding offsets does not increase total foresight-loss weight.Source: §C.3 · p. 18, Table 12 · p. 18

C.4 PREDICTION-HEAD CAPACITY

  • The 71M transformer prediction head reaches 97.45% LIBERO success, and transformers exceed MLPs by 2.40 and 0.95 points at widths 1,536 and 3,072, respectively.Larger heads do not consistently improve success.Source: §C.4 · p. 18, Table 13 · p. 18

C.5 ACTION-ONLY CONTROLS

  • Foresight raises Qwen3.5-2B LIBERO-Plus success from 77.03% to 81.14% and DINOv2-L success from 68.26% to 70.39%, with some perturbations favoring action-only training.Qwen3.5 gains are largest for background and layout; DINOv2-L gains mainly come from camera and robot-state perturbations.Source: §C.5 · p. 19, Table 14 · p. 19

C.6 SEPARATE ACTION EXPERT OR SHARED WEIGHTS

  • 81.14% LIBERO-Plus success with the separate Qwen3.5 action expert compares with 79.61% and 80.68% for shared 2B and 4B backbones, respectively.In these single-run comparisons, the expert has similar observed success to shared 4B with fewer deployed parameters; the recipe also works without a separate expert.Source: §C.6 · p. 19, Table 15 · p. 19

C.7 TRAINING BUDGET

The training-budget comparison evaluates FastWAM and JAM with Wan at 10k and 20k updates. The initialization comparison reports LIBERO success rates for pretrained and random Wan-backbone and future-anchor configurations.

C.7 TRAINING BUDGET

  • At 20k updates, JAM’s LIBERO-Plus success rises from 48.47% to 55.58%, while LIBERO success remains 97.45%.At 10k updates, JAM exceeds 20k-update FastWAM on LIBERO but trails it on LIBERO-Plus.Source: §C.7 · p. 19
  • Table 16 reports success rates for FastWAM and JAM with Wan at two training budgets, with independently trained endpoints for the 10k rows.Each run uses its own full learning-rate schedule.Source: Table 16 · p. 19

D.1 BACKBONE AND TARGET INITIALIZATION

  • Table 17 compares LIBERO success rates across pretrained and random initialization of the Wan backbone and future anchor.The action expert, policy VAE, and text conditioner remain pretrained in every configuration.Source: Table 17 · p. 20, §D.1 · p. 20

D.2 LIBERO AND LIBERO-PLUS RESULTS

On shared LIBERO-Plus tasks, paired outcomes show JAM-only successes across most perturbation categories for both Ovis-U1 and Wan. The evaluation uses a fixed 1,506-task subset, with one rollout per task.

D.2 LIBERO AND LIBERO-PLUS RESULTS

  • The shared LIBERO-Plus evaluation uses 1,506 tasks, sampled as 15% of each suite, with one rollout per task and overall success micro-averaged across tasks.The subset contains 361 Spatial, 378 Object, 389 Goal, and 378 Long tasks.Source: §D.2 · p. 20
  • Table 18 reports LIBERO success rates by Spatial, Object, Goal, and Long suites, with AVG as their four-suite mean.Published rows retain their source protocols.Source: Table 18 · p. 20
  • 71 JAM-only versus 9 baseline-only camera successes for Ovis-U1, and 45 versus 10 robot-state successes for Wan, mark each model’s largest perturbation gains.Camera outcomes for Wan are nearly balanced at 31 versus 28; both models improve in six of seven categories, with exceptions in Ovis-U1 robot-state and Wan background perturbations.Source: §D.2 · p. 20, Figure 6 · p. 21

D.3 ROBOTWIN 2.0 EVALUATION

The appendix reports RoboTwin evaluation conditions and selected matched rollouts, alongside analyses of attention, future-frame readout, and statistical significance. The comparisons describe particular tasks, representations, and fixed-checkpoint intervals rather than a new pooled benchmark result.

D.3 ROBOTWIN 2.0 EVALUATION

  • Table 19 reports RoboTwin 2.0 success rates for Clean and Randomized settings, with AVG as their mean; local evaluation covers 50 tasks and 100 trials per task and setting.Randomized scenes vary tabletop clutter, background texture, lighting, table height, and instruction wording; Clean uses the default scene.Source: Table 19 · p. 21, §D.3 · p. 21

D.4 ADDITIONAL MATCHED ROLLOUTS

  • In selected matched rollouts, JAM completes tasks where ImageWAM misses a moka-pot placement or opens the wrong drawer, and FastWAM knocks over a can or stack.LIBERO-Plus pairs share initial states, RoboTwin pairs share scene seeds, and frame times are aligned by manipulation stage.Source: §D.4 · p. 21, Figure 7 · p. 22

D.5 ACTION-TO-VISUAL ATTENTION

  • In Ovis-U1 examples, ImageWAM emphasizes the gripper and held object, while JAM emphasizes the instructed placement target.The displayed attention maps show above-mean attention; difference colors indicate higher display intensity for JAM in red and ImageWAM in blue.Source: §D.5 · p. 22, Figure 8 · p. 22

D.6 FUTURE-FRAME READOUT

  • JAM [1] has the highest mean future-frame PSNR in the readout comparison, slightly above ImageWAM and JAM [0].Frozen-policy probes reconstruct the current frame and the frame 16 steps ahead; “Current” uses the current frame as its future prediction.Source: Figure 9 · p. 23, §D.6 · p. 23

D.7 STATISTICAL SIGNIFICANCE

  • All four fixed-checkpoint confidence intervals favor JAM and exclude zero, and all supplementary exact two-sided McNemar tests have p < 0.05.The intervals use 100,000 task-bootstrap resamples; tests use matched episodes for LIBERO and matched tasks for LIBERO-Plus.Source: Table 20 · p. 23, §D.7 · p. 23

D.8 INDEPENDENT TRAINING RUNS

Three-run evaluations report low variability in LIBERO-Plus success for selected Wan and Ovis-U1 configurations. The extended related-work discussion distinguishes JAM’s training-only foresight supervision from approaches that retain prediction or generation machinery for action selection or training.

D.8 INDEPENDENT TRAINING RUNS

  • LIBERO-Plus success is 64.61 ± 0.37% for Wan and 79.92 ± 0.43% for Ovis-U1 across three independent runs per configuration.Wan uses VAE-direct [1,2] targets and 20k updates; Ovis-U1 uses pretrained Yak [1] targets and 21.7k updates.Source: §D.8 · p. 23, Table 21 · p. 23

E EXTENDED RELATED WORK

  • Unlike V-JEPA 2-AC, which uses latent rollouts for model-predictive control, JAM discards its prediction head and samples actions without rollouts or planning (Assran et al., 2025).JAM’s future targets supervise policy representations for imitation learning.Source: E EXTENDED RELATED WORK · p. 24
  • JAM supervises existing visual-backbone tokens through a separate training-only head, unlike cited predictive-policy approaches that retain added inference pathways (Zhao et al., 2026; Xia et al., 2026).It preserves the matched baseline’s action computation.Source: E EXTENDED RELATED WORK · p. 24
  • JAM removes visual generation during policy fine-tuning, unlike cited representation and foresight methods that retain generation (Qiu et al., 2026; Yu et al., 2026a).JAM also uses no predicted future features as action inputs.Source: E EXTENDED RELATED WORK · p. 24
  • The related-work discussion distinguishes frozen pretrained future anchors from EMA encoders, whose weights update during training.Table 2 and Appendix D.1 vary policy and frozen-anchor initialization.Source: E EXTENDED RELATED WORK · p. 24

Related papers

Loading 2610.37219v1…