Source-linked AI summary
ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
Mingxin Wang, Bin Hu, Bin Qian, Kaitao Jiang, Haoning Wu, Feng Yan, Bowen Jing, Ruiyang Hao, Enyi Wang, Kangning Niu, Yandan Yang, Mu Xu, Yan Wang, Houde Liu, Tianlun Li
TL;DR
Pixel-generative WAMs can hallucinate training-domain content under visual shifts, making future representation a central robustness challenge. ST-WAM combines DINOv3 semantics with VAE dynamics for future modeling and history retrieval, improving zero-shot LIBERO-Plus transfer by 21.3 percentage points and real-world success from 25.8% to 61.5%.
Problem
Pixel-generative WAMs can entangle action-relevant transitions with task-irrelevant content and drift toward training-domain visuals under shifted observations.
Method
ST-WAM uses DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining VAE dynamics through DSFE and CAIR.
Results
ST-WAM improves zero-shot LIBERO-Plus transfer over Fast-WAM by 21.3 percentage points and raises real-world success under visual shifts from 25.8% to 61.5%.
Takeaways & Limitations
Semantic-temporal modeling improves robustness under visual distribution shifts while preserving strong in-distribution performance and efficient action-only inference.
Abstract
from arXiv · showhide
World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.
Introduction
The introduction identifies Training-Distribution Hallucination as a major robustness failure of video-generative WAMs under visual shifts and motivates semantic-temporal modeling with DINOv3. It proposes ST-WAM, which combines semantic future prediction and history retrieval with VAE dynamics, improving zero-shot and real-world robustness without additional embodied pretraining.
- Problem: 70.6% of 180 audited predictions exhibited Training-Distribution Hallucination across background, illumination, and camera-viewpoint shifts.The audit covered 30 randomly sampled cases under each of three visual shifts for two representative WAMs.
- Problem: 97.6% and 98.5% success rates on LIBERO fell to 51.5% and 59.0% on LIBERO-Plus for Fast-WAM and Fast-WAM-Joint, respectively.These zero-shot drops expose a robustness limitation under visual distribution shifts.
- Representation diagnosis: DINOv3 reached 0.904 same-state cosine similarity versus 0.686 for Wan-VAE latents across 290 frame triplets.The diagnosis used same-task, same-robot-and-object-state initial frames under different visual conditions, plus a different-state reference.
- Method: ST-WAM uses DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics.DSFE jointly models future VAE latents and DINO features, while CAIR retrieves task-relevant evidence from recent DINO history using current visual-language context.
- Results: 98.7% on LIBERO and 21.3 percentage points over Fast-WAM on zero-shot LIBERO-Plus were achieved without embodied pretraining.Real-world success under visual distribution shifts increased from 25.8% with Fast-WAM to 61.5%.
Related Work
Related work spans semantic vision-language-action models, world action models, and emerging semantic or spatially structured future representations. Existing WAM approaches differ in deployment-time inference design, but their future supervision is largely rooted in pixel-generative objectives.
- Vision-Language-Action Models: VLA models use pretrained vision-language knowledge to map observations and instructions into robot actions, with newer methods adding temporal context or future-prediction objectives.Examples include IntentVLA, DreamVLA, VLA-JEPA, and DeFI.
- World Action Models: World Action Models jointly model future visual states and robot actions through video prediction and action generation.Representative methods include DreamZero, LingBot-VA, and Motus.
- World Action Models: Fast-WAM and GigaWorld-Policy omit explicit future video generation during deployment to improve inference efficiency.Their deployment designs differ from methods that couple video prediction with action generation.
- Structured Future Representations: Recent alternatives predict semantic masks, geometric-semantic cues, spatial value maps, compact latent conditions, or future states directly in DINO feature representations.These methods move beyond purely pixel-generative future objectives toward semantic or spatially structured representations.
Methodology
ST-WAM combines dual-space future modeling with current-anchored semantic history retrieval to support robust action generation under visual distribution shifts. It jointly trains visual, semantic, and action experts, while deployment reduces to an efficient action-only policy without explicit future generation.
- Dual-Space Future Experts: DSFE jointly models future dynamics in Wan2.2 VAE visual-latent and DINOv3 semantic spaces, preserving fine-grained visual dynamics while learning semantic state transitions.The visual and semantic experts use separate future prediction branches and interact through mixed attention.
- Three-Branch Mixture-of-Transformers: A three-branch Mixture-of-Transformers couples visual and semantic future experts with an action expert through layer-wise mixed attention and branch-specific flow-velocity heads.Structured cross-branch masking keeps clean current tokens as leakage-free anchors, enables mutual future refinement, and lets action tokens use current evidence from both spaces.
- Current-Anchored Intent Retrieval: CAIR uses current observation and language features as queries to retrieve task-relevant evidence from temporally embedded DINOv3 history tokens.The retrieved short-horizon intent tokens summarize recent task progress and are injected only into the action expert, guiding action generation without changing DSFE conditioning.
- Semantic-Temporal Integration: ST-WAM uses DINOv3 prospectively for future semantic supervision and retrospectively for task-relevant history retrieval, complementing pixel-generative dynamics under visual distribution shifts.DSFE supervises future semantic dynamics, whereas CAIR retrieves evidence from recent semantic history for robust action generation.
- Training and Inference: ST-WAM jointly trains visual, semantic, and action experts with flow matching, synchronizing the two future denoising stages while independently sampling the action timestep.At inference, the model reduces to an action-only policy for efficient deployment, with future flows used only during training.
Experiments
ST-WAM achieves leading in-distribution and bimanual manipulation performance, while substantially improving zero-shot and real-world robustness under visual distribution shifts. Ablations show that its gains depend on combining VAE dynamics with DINO semantic prediction and current-anchored DINO-history retrieval.
- LIBERO: 98.7% average success on LIBERO is the highest among compared methods, exceeding Motus at 97.7% and LingBot-VA at 98.5% without embodied pretraining.LIBERO evaluation covers Spatial, Object, Goal, and Long suites, totaling 40 tasks with 50 rollouts per task.
- RoboTwin 2.0: 92.77% average success on RoboTwin 2.0 is highest among compared methods, with 93.06% in clean and 92.48% in randomized settings.ST-WAM exceeds LingBot-VA at 92.20% and Fast-WAM at 91.83% without embodied pretraining.
- LIBERO-Plus: 72.8% zero-shot success on LIBERO-Plus improves Fast-WAM from 51.5% by 21.3 percentage points and remains consistent across all seven perturbation categories.The largest gains are 39.0 and 41.8 percentage points under camera and sensor-noise perturbations, respectively.
- Real-World Generalization: 61.5% success under real-world visual shifts exceeds Fast-WAM by 35.7 points, while ST-WAM drops 17.8 points from nominal to shifted conditions versus Fast-WAM’s 38.9-point drop.Under nominal conditions, ST-WAM reaches 79.3%; under compound shifts, it retains a 48.0% versus 15.3% advantage over Fast-WAM.
- Real-World Ablations: Removing the semantic future expert or CAIR reduces shifted-condition real-world performance to 41.0% and 43.7%, respectively, confirming complementary contributions to robustness.The real-world evaluation uses five tasks on an Agilex Piper 6-DoF robot with 50 demonstrations per task.
- Controlled Ablations: 72.8% LIBERO-Plus success for the full model exceeds DINO Future Only at 39.7%, Naive History Retrieval at 56.5%, and Dual-Space w/o CAIR at 66.4%.VAE History reaches 64.7% and Qwen Current Only 62.3%, showing that effective intent modeling requires explicit future-semantic prediction and current-anchored retrieval from DINO history.
Conclusion
ST-WAM addresses Training-Distribution Hallucination and improves World Action Model robustness under visual distribution shifts by combining stable future semantics with task-relevant semantic-history retrieval. Experiments show improved robustness while preserving in-distribution performance and efficient action-only inference, without large-scale embodied pretraining.
- ST-WAM targets robustness limitations exposed by Training-Distribution Hallucination under visual distribution shifts.
- DSFE combines fine-grained VAE dynamics with visually stable future semantics, while CAIR retrieves task-relevant intent from recent semantic history under current visual-language context.
- Experiments demonstrate substantially improved robustness while preserving strong in-distribution performance, efficient action-only inference, and freedom from large-scale embodied pretraining.
- Semantic-temporal modeling is identified as an effective direction for robust world-action learning beyond pixel-centric futures.
- Future work will extend ST-WAM beyond visual distribution shifts to changes in physical dynamics and embodiments.