Source-linked AI summary
How Representational Priors Shape Generalization in World Action Models
Junha Chun, Youngjoon Jeong, Taesup Kim
TL;DR
The paper examines which geometric properties of generative-backbone representations relate to WAM robustness, and how policy adaptation changes those representations. It measures nuisance orthogonality and suppression, finding that higher orthogonality is associated with stronger visual robustness after WAM training.
Problem
The paper asks which geometric properties of backbone representations support WAM generalization and how policy adaptation transforms those representations.
Source: §1 · p. 1Method
It measures nuisance orthogonality as feature displacement outside the empirical in-distribution span and analyzes nuisance suppression after policy adaptation.
Source: §1 · p. 2Results
Takeaways & Limitations
Nuisance orthogonality can serve as a diagnostic of backbone robustness before WAM policy training.
Source: §1 · p. 2Evidence is restricted to simulated LIBERO manipulation and LIBERO-Plus visual shifts; the observational results do not isolate every causal effect of orthogonality or pretraining objective.
Source: §6 · p. 9
Abstract
from arXiv · showhide
World action models (WAMs) are shown to substantially diverge under generalization performance based on their generative backbones. We investigate this gap through the representational geometry that WAMs inherit from their backbones and how policy training reorganizes it. Our central hypothesis is that a useful representational prior would structure geometry that separates nuisance information from the in-distribution manifold in advance of training as a policy. We quantify this with nuisance orthogonality; orthogonality of the feature under visual variation outside the empirical in-distribution manifold. Across various image and video backbones, we show that a higher orthogonality from the representational prior is associated with stronger robustness to visual variations after WAM training, enabling policy generalization measurable prospectively for world models. In addition, analyzing the resulting robust WAMs reveals nuisance suppression; relative suppression of feature displacement caused by such variations and robust action generation of the policy. Together, these findings provide a geometric perspective on how the representational prior of the world model shapes the generalization performance of WAMs.
1 INTRODUCTION
The paper asks which geometric properties make a generative backbone’s representational prior useful for WAM generalization and how policy adaptation changes that geometry. It introduces nuisance orthogonality and nuisance suppression to relate backbone representations and post-adaptation changes to visual robustness.
1 INTRODUCTION
- WAMs inherit representations and spatiotemporal knowledge from generative backbones, predict future observations, and generate actions from features of that reasoning process.Prior work reports improved generalization to out-of-distribution scenes through predicting how the world changes.Source: §1 · p. 1
- The paper asks which geometric structure makes a backbone’s representational prior useful for WAM generalization and how policy adaptation transforms that structure.Recent work questions whether WAM capabilities mainly reflect explicit test-time prediction or enriched representations, and whether video prediction is a useful prior or objective.Source: §1 · p. 1
- Figure 1 conceptually links feature-displacement orthogonality to generalization and depicts representation reorganization during policy adaptation through nuisance suppression.The figure is conceptual because adaptation reshapes the representation space.Source: Figure 1 · p. 2
- Higher nuisance orthogonality is associated with stronger WAM generalization across visual variations and models, supporting prospective assessment before policy training.The paper defines nuisance orthogonality as the fraction of feature displacement orthogonal to the empirical in-distribution span.Source: §1 · p. 2
- After policy adaptation, robust policies exhibit stronger and more consistent nuisance suppression, which measures representation changes and supports robust action generation.The paper describes suppression as reducing residual feature displacement from visual variations.Source: §1 · p. 2
2 RELATED WORK
Related work studies WAMs as predictive robot models, representational priors for robustness, and methods for separating nuisance from task-relevant information. This paper positions its contribution as measuring nuisance separation in pretrained representations and relating it empirically to policy robustness.
World Action Models
- Prior work reports WAM generalization and robustness under visual perturbations, while predictive world modeling can also support test-time planning by evaluating candidate actions.WAMs transfer spatiotemporal priors from generative pretraining, with future prediction linking actions to visual consequences.Source: §2 · p. 2
- Recent studies report that video prediction co-training improves world representations while explicit test-time generation adds only trivial downstream gains; image-editing representations have also aided visual generalization.These findings motivate examining inherited representational priors and how they relate to robustness after policy adaptation.Source: §2 · p. 3
Nuisance separation and representation geometry
- Nuisance invariance, disentanglement, adversarial removal, and orthogonality objectives have been studied for separating task information from irrelevant variation and for domain adaptation and fair representation learning.Unsupervised disentanglement is not identifiable without inductive biases or supervision (Locatello et al., 2019).Source: §2 · p. 3
- Prior robot-manipulation and visual-reinforcement-learning work relates pretrained representations to robustness across shifts; this study instead measures nuisance separation in pretrained representations and relates it empirically to policy robustness.Related approaches also separate nuisance factors in latent-action pretraining.Source: §2 · p. 3
3 EXPERIMENTAL SETUP
The experiments motivate image-editing backbones as a source of representations for visual shifts, while noting that pretraining objective alone does not explain backbone comparisons. Evaluation uses LIBERO-Plus visual shifts and WAM success rates across several generative backbones.
3.1.1 WHY IMAGE EDITING COULD BE BENEFICIAL FOR GENERALIZATION
- Image editors can zero-shot edit LIBERO data, and their instruction-conditioned denoising learns relationships between reference scenes and edited targets across visual changes.The paper contrasts this with video-generation models’ bias toward temporal continuity, motion, and dynamics rather than counterfactual world changes.Source: Figure 2 · p. 3, §3.1.1 · p. 3
3.1.2 REPRESENTATIONAL PRIOR IS MORE THAN PRETRAINING OBJECTIVE
- FLUX.2 initialized from scratch reaches 63.9%, compared with 46.5% for Wan2.2 and FastWAM, so image-editing pretraining alone cannot explain the comparison.The paper treats representational priors as reflecting design choices including tokenizers, architecture, initialization, language injection, and pretraining data.Source: §3.1.2 · pp. 3–4, Table 1 · p. 4
3.2 EXPERIMENTAL SETTING
- The diagnostic pairs clean and visually shifted LIBERO-Plus observations sharing simulator states across lighting, background, sensor noise, and camera viewpoint at multiple strengths.Forty clean references estimate the empirical in-distribution feature subspace; 1,200 paired shifted observations measure variation-induced displacement. Generalization in language understanding, task composition, and action semantics is outside scope.Source: §3.2 · p. 4
- WAM success rates are evaluated on LIBERO-Plus after training on LIBERO with configurations from FastWAM and ImageWAM, using five generative backbones and 5,418 evaluation trajectories.The study additionally extends FlashMotion, Kiwi-Edit, SANA-Video 2.0 5B, and MAGI-1 4.5B to WAMs with a MoT architecture and action expert.Source: §3.2 · p. 4
4 NUISANCE ORTHOGONALITY
The section examines whether feature displacements under visual variation are geometrically separated from the in-distribution representation manifold. It defines nuisance orthogonality and reports that it aligns with WAM generalization, including across additional generative backbones.
4 NUISANCE ORTHOGONALITY
- The section frames visual-variation displacement direction and its orthogonality to the in-distribution manifold as a way to assess representational disentanglement.Figure 4 reports that nuisance orthogonality is consistent across retained-rank changes.Source: §4 · pp. 4–5, Figure 4 · p. 6
4.1 DEFINITION
- Nuisance orthogonality describes how feature displacement from visual variation relates to the empirical in-distribution manifold, with high scores indicating mostly orthogonal displacement.The analysis extracts clean and transformed features from backbone layers and aggregates the orthogonality over layers and the diagnosis dataset.Source: §4.1 · p. 5
4.2.1 NUISANCE ORTHOGONALITY ALIGNS WITH GENERALIZATION
- With full rank r = 39 for 40 reference scenes, the results show a general tendency for higher nuisance orthogonality to align with stronger LIBERO-Plus generalization, though sensitivity varies by visual variation and difficulty.Orthogonality remains consistent as retained rank changes, and its value increases as rank decreases because fewer directions contribute to projected displacement.Source: §4.2.1 · p. 6
4.2.2 IMPLICATIONS FOR MODEL SELECTION
- Across additional generative backbones, nuisance orthogonality measured before WAM training predicts generalization performance, and video models with high generalization support useful priors across pretraining objectives.The models analyzed include FlashMotion, Kiwi-Edit, SANA-Video 2.0 5B, and MAGI-1 4.5B.Source: §4.2.2 · p. 6
5 NUISANCE SUPPRESSION
The section measures how WAM adaptation changes visual-variation responses within and outside the in-distribution feature span. ImageWAM shows more consistent nuisance suppression than FastWAM, including in camera-shift visualizations.
5.1 DEFINITION
- The clean in-distribution observations define a checkpoint-specific covariance basis and variances; feature displacement is measured between each clean observation and its transformed version.The same reference observations are used for both checkpoints, but each checkpoint has its own fitted basis and variances.Source: §5.1 · p. 6
- For each transformation, the section compares standardized responses within the in-distribution span and in its orthogonal component, using WAM/WM ratios to indicate relative suppression below 1.The orthogonal response is normalized by total variance of the clean reference features.Source: §5.1 · pp. 6–7
5.2 ANALYSIS
- ImageWAM suppresses both response components across tested variations, whereas FastWAM shows weaker, less consistent suppression and insufficient camera suppression alongside 16.4% camera-perturbation success.The WAM/WM ratios fall below 1 for ImageWAM in both components; the comparison reports stronger robustness for ImageWAM.Source: §5.2 · pp. 7–8
- The controlled camera experiment compares 64 intermediate views over weak and strong perturbation ranges on 10 scenes, with representations visualized using 3D PCA.The camera ranges are [−14°, +14°] and [−56°, +56°].Source: §5.2 · p. 8, Figure 6 · p. 8
- Across weak camera shifts, ImageWAM’s perturbed views contract near distinct original views, while FastWAM separates camera trajectory from scene identity; under strong shifts, FastWAM becomes camera-dominated.ImageWAM’s representation paths still vary with camera viewpoint under strong shifts, while scenes remain distinguishable.Source: §5.2 · p. 8
- ImageWAM’s in-distribution variance rises 13.4-fold while its within-span and orthogonal displacements rise 2.73 and 2.48; FastWAM’s corresponding values are 4.42, 5.63, and 3.18.These reported changes show stronger relative suppression for ImageWAM than FastWAM.Source: §5.2 · p. 9
6 CONCLUSION
The conclusion links generative-backbone geometry to visual generalization in WAMs. It identifies simulated LIBERO manipulation and LIBERO-Plus visual shifts as the evidence scope.
6 CONCLUSION
- Across evaluations, higher nuisance orthogonality is generally related to stronger WAM robustness in LIBERO-Plus, and additional world-model evaluations support examining geometry beyond image-versus-video pretraining.Orthogonality measures whether action-preserving visual changes displace features outside the empirical subspace of in-distribution observations.Source: §6 · p. 9
- After adaptation, robust ImageWAM exhibits consistent nuisance suppression, while FastWAM shows weaker and less consistent suppression.The conclusion characterizes useful priors as separating visual nuisance from in-distribution samples, with adaptation reducing nuisance’s relative influence.Source: §6 · p. 9
- The evidence is limited to simulated LIBERO manipulation and LIBERO-Plus visual shifts, excluding language-instruction and action-compositionality benchmarks; the experiments are observational and do not isolate every causal effect.The study examines representational geometry alongside compounding effects and generalization performance.Source: §6 · p. 9
A.1 MODEL AND TRAINING CONFIGURATIONS
The appendix describes baseline and additional WAM configurations and model-specific extensions. It reports training resources and summarizes model specifications in Tables 2 and 3.
A.1 MODEL AND TRAINING CONFIGURATIONS
- Tables 2 and 3 summarize model specifications and additional-backbone training configurations, respectively, while the baseline and additional WAMs follow FastWAM and ImageWAM settings.The supplied table captions identify their scope but do not provide individual specification values.Source: Table 2 · p. 14, Table 3 · p. 14, §A.1 · p. 14
- FlashMotion and Kiwi-Edit extend adapted Wan2.2-TI2V-5B using FastWAM’s action-expert design, with preprocessed ControlNet weights and ZeroConv adaptors for FlashMotion’s trajectory conditioning.The paper attributes FlashMotion’s larger action expert to this design.Source: §A.1 · pp. 14–15
- SANA-Video 2.0 and MAGI-1 use ImageWAM’s action-expert design with 30 action-expert layers; training runs used four Nvidia H100s and took up to 38 hours per WAM.The additional models use ImageWAM’s full IDM attention strategy rather than their original autoregressive or block-causal generation strategies.Source: §A.1 · p. 15
A.2 ZERO-SHOT EDITING
The zero-shot editing setup uses prompts generated from original LIBERO and LIBERO-Plus samples to produce edited scenes, including material, lighting, and blur changes.
A.2 ZERO-SHOT EDITING
- Prompts generated with Claude Opus 5 xhigh from original LIBERO and LIBERO-Plus samples were used zero-shot with image-editing models, at CFG 3.0 and 50 inference steps.Source: §A.2 · p. 15
- One prompt replaces the tabletop with cobblestone texture and the wall with blue plaster while preserving scene objects, geometry, positions, camera, and lighting.Source: §A.2 · p. 15
- Another prompt changes only illumination, specifying dimmer cool-blue and subtle magenta casts, softer contrast and shadows, and an approximately 0.70x mean-light ratio.Source: §A.2 · p. 15
- A separate prompt applies moderate Gaussian defocus blur.Source: §A.2 · p. 15