CSWAM: Better Causal Semantic Representations for Out-of-Distribution Generalization in World Action Models

• FastWAM success falls sharply under visual shifts, while reconstruction-oriented VAE latents preserve appearance details without explicit alignment to task-relevant semantics.Background or object-appearance changes can alter visual context even when manipulation state is unchanged.Source: §1 · pp. 1–2
• Without observation history, the action expert cannot use temporal evidence about motion, state transitions, or task progress to guide action prediction.The paper identifies this, alongside unstable semantics, as a reason policies may rely on training-scene-specific correlations.Source: §1 · p. 2
• With embodied pretraining, CSWAM raises RoboTwin 2.0 Clean-to-Randomized Randomized success from 10.16% to 45.18%, a 35.02-point gain over FastWAM.Across two real-robot tasks and three OOD difficulty levels, average success rises from 27.5% to 70.0%.Source: §1 · p. 2
• CSWAM uses a frozen V-JEPA 2.1 encoder and a causal semantic expert to predict future semantic features from sparse current and past observations.The history-derived context is shared with video and action streams, while action inference omits future target streams and denoises only actions.Source: §1 · p. 2
• The paper evaluates temporally predictive representations and observation history through controlled visual-shift evaluations in simulation and on real robots.The authors report that CSWAM improves both standard manipulation performance and OOD generalization over FastWAM.Source: §1 · p. 2

2.1 WORLD ACTION MODELS

• FastWAM jointly trains video and action prediction but removes future-video denoising at deployment; CSWAM retains action-only inference while adding temporally predictive V-JEPA history.Other efficiency trade-offs include asynchronous future-latent prediction and compact future representations reused during action denoising.Source: §2.1 · p. 3

2.2 VISUAL REPRESENTATIONS FOR ROBOT MANIPULATION

• Reconstruction-oriented VAE latents preserve appearance details, while framewise DINO features do not explicitly model temporal relations; CSWAM predicts future V-JEPA features from sparse history.V-JEPA 2.1 improves spatial grounding through dense context loss and deep self-supervision (Mur-Labadia et al., 2026).Source: §2.2 · p. 3

2.3 OOD GENERALIZATION IN ROBOT MANIPULATION

• Prior work reports visual-shift sensitivity in visuomotor policies and VLAs, while CSWAM evaluates Clean-to-Randomized transfer and compounded real-robot shifts.Related WAM approaches use semantic-history retrieval or retain compact future representations during action denoising.Source: §2.3 · pp. 3–4

3.1 OVERVIEW

• CSWAM augments the video–action architecture with a semantic stream derived from current and past observations, while deployment predicts action chunks from observed features.The deployed policy uses the current video representation and frozen semantic representation of the observation history.Source: §3.1 · p. 4

3.2 CAUSAL SEMANTIC EXPERT

• A frozen V-JEPA 2.1 encoder represents sparse observation history, and the semantic expert learns to extrapolate future semantic features from it.Temporal slices aggregate short local transitions, while attention integrates evidence across the full history.Source: §3.2 · pp. 4–5
• A learnable spatiotemporal projection reduces the dense V-JEPA feature grid’s spatial token count while preserving temporal order before semantic-expert processing.Source: §3.2 · p. 5

3.3 CROSS-STREAM CONTEXT INTEGRATION

• Masked Mixture-of-Transformers attention lets video tokens use semantic history and lets the action expert attend to both current video and semantic-history tokens.Future video and semantic streams can also exchange information during training.Source: §3.3 · pp. 5–6

3.4 JOINT TRAINING AND ACTION-ONLY INFERENCE

• Joint training couples separate video, semantic, and action prediction objectives, while inference discards future-video and future-semantic streams and iteratively denoises actions.The video and semantic streams share a sampled noise level; the action stream samples independently.Source: §3.4 · p. 6

4.1 EVALUATION PROTOCOLS AND SETUP

• The protocols cover 50 RoboTwin 2.0 tasks, seven LIBERO-Plus shifts, and two real-robot tasks evaluated across three OOD levels.RoboTwin2.0-Clean2Random trains only on Clean demonstrations and tests on Randomized scenes without adaptation; the real-robot tasks use 12 hours of Sort and 4 hours of Fold fine-tuning.Source: §4.1 · p. 6

4.2 SIMULATION: STANDARD PERFORMANCE AND OOD TRANSFER

• With embodied pretraining, CSWAM reaches 64.82% average success on Clean2Random versus 43.42% for FastWAM, including a 35.02-point Randomized gain.Without embodied pretraining, CSWAM also exceeds FastWAM by 11.17 points in average success.Source: Table 3 · p. 7, §4.2 · p. 7
• On RoboTwin2.0-Full, CSWAM achieves 94.5% average success, 2.6 points above FastWAM, under mixed Clean and Randomized training.Because both settings appear in training, this evaluates mixed-training robustness rather than OOD transfer.Source: §4.2 · p. 7, Table 2 · p. 8
• On LIBERO-Plus, CSWAM achieves 77.1% overall success, 4.3 percentage points above ST-WAM, with robot-initial-state shifts its most challenging category.Its success reaches 85.9% under camera-viewpoint shifts and 93.4% under lighting shifts.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 REAL-WORLD OOD GENERALIZATION

• Across two real-robot tasks and three OOD difficulty levels, CSWAM raises average success from 27.5% to 70.0%.The advantage persists under Normal, Hard, and Extreme shifts, though Extreme Fold remains challenging for both methods.Source: Table 5 · p. 8, §4.3 · pp. 8–9

4.4 ABLATION STUDIES

• Removing observation history reduces Clean and Randomized success by 1.7 and 1.8 points, while removing the video expert causes larger drops of 6.1 and 8.1 points.The authors attribute the history benefit to temporal evidence and the video-expert benefit to fine-grained visual dynamics.Source: §4.4 · p. 9, Table 6 · p. 9

4.5 COMPONENT ANALYSIS

• Replacing V-JEPA 2.1 with DINOv3 lowers Clean and Randomized success by 1.4 and 1.5 points despite DINOv3 having approximately 3.5× more parameters.The comparison matches downstream token budgets and supports jointly encoded temporal features over independent image descriptors.Source: Table 7 · p. 9, §4.5 · p. 9
• Increasing history from 4 to 8 frames adds 1.1 Clean and 1.3 Randomized success points, following smaller gains from 0 to 4 frames.The larger second gain indicates that four frames do not saturate useful temporal context.Source: §4.5 · p. 9

5 CONCLUSION

• CSWAM raises Clean2Random OOD success from 10.16% to 45.18% over FastWAM and improves average success from 10% to 55% across Hard and Extreme real-robot settings.The conclusion also reports 94.5% average success on RoboTwin2.0-Full.Source: §5 · p. 9
• CSWAM augments an action-only WAM with V-JEPA 2.1 representations of causal observation history and predictive semantic modeling.The conclusion reports that semantic and generative video dynamics remain complementary.Source: §5 · p. 9