PaperlayerView exact revision on arXiv

Source-linked AI summary

What Visual Representations Do World Action Models Need?

Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang

arXiv:2610.13024v1cs.CVcs.RO

TL;DR

WAMs inherit VAE representations optimized for reconstruction, while evidence remains limited on which visual features and integration mechanisms best support manipulation. The paper systematically compares representations, integration mechanisms, and feature combinations, then instantiates its findings as R2-WAM. General-purpose representations and channel-wise joint prediction emerge as useful design choices, and R2-WAM improves on VAE-only baselines across two benchmarks.

  • Problem

    WAMs inherit VAE representations optimized for reconstruction, and systematic evidence remains limited on which visual representations and integration mechanisms best support manipulation.

    Source: §1 · pp. 1–2

  • Method

    The study compares visual representation sources, five integration mechanisms, and feature combinations under a common WAM backbone and shared training protocol.

    Source: §1 · pp. 2–3

  • Results

    General-purpose representations provide broader robustness gains, channel-wise joint prediction yields the largest average gains across matched sources, and R2-WAM improves over VAE-only baselines by 5.9 and 2.2 percentage points on RoboCasa GR1 Tabletop and RoboTwin 2.0, respectively.

    Source: §1 · pp. 2–3, §4.5 · p. 9

  • Takeaways & Limitations

    The practical recipe is to start with a strong general-purpose representation, integrate it through channel-wise concatenation, and retain additional sources when they improve task success in the intended combination.

    Source: §5 · p. 10

Abstract

from arXiv · show

World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R$^2$-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.

1 INTRODUCTION

WAMs inherit video-generation priors and VAE representations optimized for reconstruction, whose support for manipulation is unclear. The paper compares representation sources, integration mechanisms, and feature compositions under a shared protocol to derive design guidance and validate R2-WAM.

1 INTRODUCTION

  • The inherited VAE is optimized for visual reconstruction, but whether its representations support manipulation remains unclear.Its reconstruction objective does not explicitly prioritize manipulation-relevant information such as object semantics and spatial relationships.Source: §1 · p. 1
  • Existing studies leave representation choice and integration mechanism difficult to disentangle, while the benefits of combining sources remain an empirical question.The study addresses these gaps through controlled comparisons under a shared WAM backbone and training protocol.Source: §1 · p. 2
  • The study finds that general-purpose visual representations can match or outperform specialized ones while providing broader robustness gains.Channel-wise joint prediction yields the largest average gains across matched sources, while additional sources can help one combination but hurt another.Source: §1 · pp. 2–3
  • R2-WAM instantiates the resulting representation recipe and is validated across WAM formulations, benchmarks, and real-world robot experiments.The recipe is motivated by the controlled study's comparisons of representation sources, five integration mechanisms, and feature combinations.Source: Figure 1 · p. 2, §1 · p. 3

2 RELATED WORK

Prior WAM work integrates visual representations in diverse ways, while visual encoders capture overlapping capabilities shaped by their pretraining objectives. Systematic comparisons are needed to establish how those features translate into manipulation gains.

2 RELATED WORK

  • Prior WAM studies use specific visual representations within distinct designs, but systematic comparisons are still needed to identify complementary features.Examples include feature-space dynamics, semantic subgoals, visual–action tokenizers, future semantic features, and joint prediction of multiple visual streams.Source: §2 · p. 3
  • Visual encoders target self-supervision, language alignment, segmentation, geometry, or interaction, and their capabilities can overlap.How these pretraining emphases translate into WAM manipulation gains remains an empirical question.Source: §2 · p. 3

3 STUDY DESIGN

The study uses a VAE-based WAM with video and action experts, evaluates frozen external visual encoders, and compares five mechanisms for integrating their features.

3.1 BASE WAM ARCHITECTURE

  • The base WAM uses video and action experts to jointly predict future-video latents and an action chunk, with asymmetric joint attention and shared language conditioning.Training combines video and action flow-matching losses, and inference jointly denoises future-video latents and actions.Source: §3.1 · pp. 3–4

3.2 VISUAL REPRESENTATIONS

  • Thirteen frozen visual encoders are compared, with pretraining objectives used to indicate captured information and to classify sources as general-purpose or specialized.The encoders extract features from the RGB observations available to the base WAM.Source: §3.2 · p. 4

3.3 FEATURE INTEGRATION MECHANISMS

  • The five integration mechanisms span representation alignment, action conditioning, and joint prediction, differing in feature fusion and Transformer parameter sharing.They include REPA, gated cross-attention, channel-wise concatenation, token-wise concatenation, and a separate feature expert; feature compositions use spatially aligned, channel-concatenated features.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Across controlled WAM comparisons, general-purpose visual representations provide broad robustness gains, channel-wise concatenation performs best on average, and the value of additional sources depends on the existing features. The selected representation recipe also improves benchmark and real-robot performance.

4.1 EXPERIMENTAL SETUP

  • LIBERO comparisons hold training data and key model and optimization settings fixed while varying representations, integration mechanisms, or source combinations.LIBERO uses 50 rollouts per task; LIBERO-Plus is evaluated without further training using three seeds and all 10,030 episodes per seed across seven perturbation categories.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 RQ1: WHICH VISUAL REPRESENTATIONS IMPROVE WAM PERFORMANCE?

  • 82.2% vs. 80.0% is the average success rate for general-purpose versus specialized representations, and general-purpose sources lead in all seven perturbation categories.The comparisons use LIBERO-Plus success rates under channel-wise concatenation.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 RQ2: WHICH FEATURE INTEGRATION MECHANISM YIELDS THE GREATEST PERFORMANCE GAINS FROM EXTERNAL FEATURES?

  • Across thirteen visual representations, channel-wise concatenation achieves the highest average LP success rate at 81.5%, exceeding the next-best integration mechanism by 5.1 percentage points.Future-only prediction reaches 81.9%, close to the full configuration at 84.0% and above current-only conditioning at 76.5%.Source: §4.3 · p. 7

4.4 RQ3: DOES COMBINING MORE FEATURE SOURCES CONSISTENTLY IMPROVE WAM PERFORMANCE?

  • DINOv3 alone adds 9.3 percentage points over VAE-only, while other sources add at most a further 3.7 points among evaluated combinations.The benefit of an added source depends on the existing representations; the study recommends selecting additions for complementary task-relevant information.Source: §4.4 · p. 8

4.5 REPRESENTATION RECIPE AND VALIDATION

  • R2-WAM reaches 84.7% success on RoboCasa GR1 Tabletop and 92.8% on RoboTwin 2.0, improving over VAE-only baselines by 5.9 and 2.2 percentage points, respectively.The paper reports state-of-the-art performance on RoboCasa GR1 Tabletop.Source: §4.5 · p. 9, Table 3 · p. 9
  • R2-WAM improves complete-task success by 10.0 percentage points on each of three real-robot tasks; tool-sweeping debris collection rises from 48.3% to 66.7%.The three complete-task success measures are for air-fryer food preparation, instructed pick-and-place, and cup pushing.Source: Table 2 · p. 9, Figure 5 · p. 9, §4.5 · p. 10

5 CONCLUSION

The conclusion recommends starting with a strong general-purpose representation, integrating it through channel-wise concatenation, and retaining other sources when they improve task success in the intended combination. The appendix specifies the visual encoder configurations and features used.

5 CONCLUSION

  • R2-WAM applies a recipe of starting with a strong general-purpose representation, using channel-wise concatenation, and retaining additional sources when they improve success in the intended combination.The paper reports gains across the evaluated robot embodiments and environments.Source: §5 · p. 10

A VISUAL REPRESENTATION CONSTRUCTION

B FEATURE INTEGRATION IMPLEMENTATION DETAILS

The implementation compares three WAM formulations and five feature-integration mechanisms under shared settings. The mechanisms differ in how features enter the model and whether future features are predicted or used only during training.

B.1 WAM FORMULATIONS

  • Joint video–action denoising gives the action expert access to current visuals, noisy future visuals, and actions; direct prediction uses current visuals only, while IDM conditions on future visuals.For IDM, future visuals are ground truth during training and predicted before action denoising at inference; direct prediction generates no future visuals at inference.Source: §B.1 · p. 14

B.2 INTEGRATION MECHANISMS

  • REPA aligns a projected video-expert state from block eight with teacher features over valid future tokens, excluding the observed slice; its teacher and projector are training-only.Source: §B.2 · p. 14
  • Gated cross-attention uses the post-mixed-attention action state as query and current-observation features as keys and values, while retaining language cross-attention.Each layer has a scalar gate initialized to zero; the added branch has no auxiliary feature loss and remains active at inference.Source: §B.2 · pp. 14–15
  • Channel-wise concatenation aligns external features with the VAE latent grid, preserving the video token grid and attention sequence length while jointly predicting future latents and features.At inference, future features participate in joint denoising, but only VAE latents are decoded.Source: §B.2 · p. 15
  • Token-wise concatenation appends flattened feature tokens to VAE tokens; they share video-expert Transformer blocks and use a separate feature prediction head.The attention mask lets action tokens attend to observed and noisy future visual streams, but visual tokens do not attend to actions.Source: §B.2 · pp. 15–16
  • The feature-expert mechanism adds an independent 30-block Transformer expert, and its features participate in joint denoising at inference.The expert has hidden width 1024; the observed feature prefix is restored after each solver step and the predicted feature trajectory is discarded.Source: §B.2 · p. 16
  • Across four REPA depth-and-weight configurations, mean LIBERO-Plus success varies by less than 0.1 percentage points, and increasing alignment weight does not improve the mean.The sensitivity experiments use five visual encoders.Source: §B.2 · p. 16, Table 5 · p. 17
  • The five integration mechanisms are compared using a common WAM backbone and training budget.Within the main LIBERO comparisons, training settings are held fixed as the representation source, mechanism, or source set varies.Source: §B.2 · p. 16, §C.1 · p. 17

C.2 SIMULATION EVALUATION AND REPORTING

The appendix defines benchmark coverage and aggregation, then describes the real-robot setup and evaluation measures. The four physical tasks are evaluated with 20 trials per model per task.

C.2 SIMULATION EVALUATION AND REPORTING

  • RoboCasa GR1 Tabletop uses 50 trials per task across 24 tasks, while RoboTwin 2.0 uses 100 trials per task across 50 tasks in clean and randomized settings, weighted equally.LIBERO uses 50 rollouts per task across 40 tasks; a complete LIBERO-Plus evaluation has 10,030 episodes across seven perturbation categories.Source: §C.2 · p. 17
  • Evaluation removes appended perturbation descriptions from official LIBERO-Plus instructions, except that rewritten instructions are retained for the language-perturbation category.Source: §C.2 · p. 17
  • One trained checkpoint per configuration is evaluated; RQ1 and RQ3 report means and sample standard deviations across three evaluation seeds.Source: §C.2 · p. 17

C.3 REAL-ROBOT TRAINING AND EVALUATION

  • The four real-robot tasks use a Galbot G1 with two arms, grippers, and one head-mounted plus two wrist-mounted cameras; actions are joint positions and gripper states.Models use 200 demonstrations each for air-fryer preparation and instructed pick-and-place, and 100 each for tool sweeping and cup pushing; they train for 10 epochs.Source: §C.3 · p. 18
  • The VAE-only WAM and R2-WAM are evaluated for 20 trials per task; three tasks use complete-task success, while tool sweeping uses mean debris collection rate.Each sweeping trial starts with six debris pieces, and collection rate measures partial rather than binary task completion.Source: §C.3 · p. 18

D.1 FULL SOURCE–MECHANISM RESULTS

The appendix reports detailed LIBERO and LIBERO-Plus results for representation–mechanism pairs, and describes a DINOv3 ablation that varies current-feature conditioning and future-feature prediction. The supplied passages establish the reporting scope and ablation setup, not numerical outcomes.

D.1 FULL SOURCE–MECHANISM RESULTS

  • Table 8 covers LIBERO averages and LIBERO-Plus perturbation-level and overall success rates for thirteen representations across five integration mechanisms.LIBERO is reported as a four-suite average; LP Total is the full-benchmark micro-average success rate.Source: Table 8 · p. 19, §D.1 · pp. 19–20

D.2 CURRENT-FEATURE CONDITIONING AND FUTURE-FEATURE PREDICTION

  • The DINOv3 ablation compares current-only, future-only, and full configurations under the same attention-mask rule, training duration, remaining loss weights, and evaluation settings.Current-only removes future-feature channels and prediction loss; future-only omits current-frame features but retains future-feature supervision and denoising.Source: §D.2 · p. 20, Table 9 · p. 21
  • The feature-prediction visualization projects DINOv3 features onto three principal components mapped to RGB and overlays the maps on ground-truth future images.The reference and predictions use a shared PCA projection and color scale.Source: §D.2 · p. 21

D.3 LOCAL ACTION SENSITIVITY ANALYSIS

The analysis measures how strongly action outputs respond locally to current external visual features. It also specifies how future external-feature states are constructed and how candidate sources are added progressively.

D.3 LOCAL ACTION SENSITIVITY ANALYSIS

  • The derivative tracks raw current-frame DINOv3 features through the full ten-step denoising process, with VAE latents and other inputs fixed.It measures total dependence on current external-feature input, not dependence on a generated future-feature state.Source: §D.3 · p. 21
  • Figure 3(d) plots scores across nine fixed observation states; points vary across states, not independent training replications or confidence intervals.Colors mark translation, rotation, and gripper outputs; the horizontal axis is logarithmic, while vertical offsets have no quantitative meaning.Source: §D.3 · p. 22
  • Larger scores indicate stronger local action sensitivity in the measured feature coordinates.The interpretation holds other inputs and sampling noise fixed and considers a small feature change to first order.Source: §D.3 · p. 22

E.1 FEATURE CONSTRUCTION AND TRAINING OBJECTIVE

  • Future VAE latents and external features are initialized from noise and jointly denoised, while current features remain clean and are excluded from feature losses.Each source has a loss on valid future targets, and source processing and concatenation order remain fixed between training and evaluation.Source: §E.1 · p. 22

E.2 PROGRESSIVE COMPOSITION AND CONDITIONAL GAINS

  • Composition tests begin with DINOv3, then add one candidate source to DINOv3-based parent sets, including a three-source parent for the four-source group.The seven candidates are V-JEPA 2.1, SigLIP 2, SAM 3, VGGT-Ω, HRP, UniTok, and C-RADIOv4.Source: §E.2 · p. 22

E.3 PER-SEED RESULTS AND EVALUATION SETTINGS

The composition evaluations use a fixed checkpoint and a three-seed full-benchmark protocol, with specified inference settings.

E.3 PER-SEED RESULTS AND EVALUATION SETTINGS

  • Each composition uses one fixed trained checkpoint and three evaluation seeds, each covering all 10,030 LP episodes.The mean is the average of the three full-benchmark success rates, with sample standard deviation.Source: §E.3 · p. 22
  • Evaluations use ten inference steps, replan every ten steps, wait 30 initial steps, and omit action ensembling.Text guidance scale is 1.0 and gripper outputs are binarized.Source: §E.3 · p. 22
  • Table 10 reports RQ3 composition success rates on LIBERO-Plus across three evaluation seeds, with percentages and evaluation-seed standard deviations.The standard deviation reflects seed variability for a fixed trained checkpoint.Source: Table 10 · p. 23

F TASK-LEVEL ROBOCASA RESULTS

Table 11 reports task-level R2-WAM success rates on RoboCasa GR1 Tabletop, with 50 trials per task.

F TASK-LEVEL ROBOCASA RESULTS

  • RoboCasa GR1 Tabletop results are reported as task-level success rates.Source: Table 11 · p. 23
  • Each RoboCasa GR1 Tabletop task is evaluated with 50 trials.Source: Table 11 · p. 23
  • Table 11 presents the R2-WAM results on RoboCasa GR1 Tabletop.Source: Table 11 · p. 23

G TASK-LEVEL ROBOTWIN RESULTS

Table 12 reports task-level R2-WAM success rates on RoboTwin 2.0, with 100 trials per task in both clean and randomized settings.

G TASK-LEVEL ROBOTWIN RESULTS

  • RoboTwin 2.0 results are reported as task-level success rates for clean and randomized settings.Source: Table 12 · p. 24
  • Each of the 50 RoboTwin 2.0 tasks is evaluated with 100 trials in each setting.Source: Table 12 · p. 24
  • Table 12 presents the R2-WAM task-level results on RoboTwin 2.0.Source: Table 12 · p. 24
Loading 2610.13024v1…