Source-linked AI summary
S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight
Haodong Yan, Zhide Zhong, Jiaguan Zhu, Junjie He, Weilin Yuan, Wenxuan Song, Xin Gong, Yingjie Cai, Guanyi Zhao, Xu Yan, Bingbing Liu, Ying-Cong Chen, Haoang Li
TL;DR
Existing VAMs must trade real-time inference against high-fidelity foresight because multi-step generation is slow and one-step features are noisy. S-VAM distills geometric and semantic representations from the diffusion model’s own multi-step videos into one-step inference, and experiments show state-of-the-art performance with efficient, precise manipulation.
Problem
Current video-action models cannot simultaneously provide real-time inference and high-fidelity foresight because multi-step generation is slow while one-step features are noisy and entangled.
Method
S-VAM uses lightweight decouplers and self-distillation to map noisy one-step diffusion features to DPAv3 geometric and DINOv2 semantic targets extracted from multi-step generated videos.
Results
S-VAM achieves state-of-the-art performance on simulated benchmarks and enables efficient and precise manipulation in complex environments.
Takeaways & Limitations
S-VAM provides a single-step shortcut that retains high-fidelity planning while maintaining efficient inference for robotic manipulation.
Takeaways & Limitations
Alternative motion-aware targets from VideoMAEv2 and V-JEPA2 produce sub-optimal results, possibly because current video models have lower fine-grained feature fidelity than specialized image models.
Abstract
from arXiv · showhide
Video action models (VAMs) have emerged as a promising paradigm for robot learning, owing to their powerful visual foresight for complex manipulation tasks. However, current VAMs, typically relying on either slow multi-step video generation or noisy one-step feature extraction, cannot simultaneously guarantee real-time inference and high-fidelity foresight. To address this limitation, we propose S-VAM, a shortcut video-action model that foresees coherent geometric and semantic representations via a single forward pass. Serving as a stable blueprint, these foreseen representations significantly simplify the action prediction. To enable this efficient shortcut, we introduce a novel self-distillation strategy that condenses structured generative priors of multi-step denoising into one-step inference. Specifically, vision foundation model (VFM) representations extracted from the diffusion model's own multi-step generated videos provide teacher targets. Lightweight decouplers, as students, learn to directly map noisy one-step features to these targets. Extensive experiments in simulation and the real world demonstrate that our S-VAM outperforms state-of-the-art methods, enabling efficient and precise manipulation in complex environments. Our project page is https://haodong-yan.github.io/S-VAM/
1 Introduction
S-VAM addresses the trade-off between fast but noisy one-step features and precise but slow multi-step video generation. It distills geometric and semantic foresight into a single forward pass for efficient, precise manipulation.
- Motivation: VLA models lack spatiotemporal foresight and depend on costly robot action data because their underlying vision-language models are trained mainly on static image-text pairs.VAMs instead exploit predictive priors from abundant internet-scale videos to reduce reliance on robot demonstrations.
- Motivation: Multi-step video generation provides detailed planning priors but is too slow for high-frequency control, while one-step features are fast but noisy, entangled, and weak in geometry and semantics.The missing cues include geometry for monocular-depth ambiguity and semantic distinctiveness for separating task-relevant objects.
- S-VAM: S-VAM foresees coherent geometric and semantic representations in one forward pass to guide precise action generation.Lightweight decouplers explicitly disentangle noisy one-step diffusion features into specialized representations.
- S-VAM: Self-distillation trains decouplers as students using DPAv3 geometry and DINOv2 semantic representations extracted from the diffusion model’s own multi-step generated videos.This condenses structured multi-step denoising priors into one-step inference.
- Results: S-VAM outperforms state-of-the-art methods, enabling efficient and precise manipulation in complex scenarios.The paper reports this outcome across its evaluated settings.
2 Related Works
Related work develops video-action pipelines for visual foresight and vision foundation model features for semantic and geometric understanding. These approaches motivate S-VAM’s combination of predictive video representations with specialized foundation-model targets.
- Vision-Language-Action Models: VLA methods map text instructions and visual observations to actions but struggle with spatiotemporal foresight because their pretrained vision-language models primarily use static image-text data.This motivates methods that incorporate video-based predictive representations.
- Video-Action Models: Video-action models pair video diffusion models that synthesize future visual plans with action experts that predict control signals.The paradigm leverages visual foresight from video generation for robotic action learning.
- Video-Action Models: Multi-step video generation improves visual foresight but incurs prohibitive inference latency, motivating methods that directly extract one-step denoising features.This establishes the efficiency-fidelity trade-off addressed by S-VAM.
- Vision Foundation Models: Vision foundation models provide fine-grained semantic context, while related approaches add explicit depth or point-cloud cues for geometric understanding.S-VAM differs by distilling geometric and semantic foresight rather than explicitly injecting these cues at the input.
3 Method
S-VAM uses a video diffusion backbone, decouples one-step features into future geometric and semantic representations, and fuses them with diffusion features to condition action prediction.
- Overview: S-VAM foresees geometric and semantic representations for action generation without multi-step denoising.The method is organized around preliminaries, geometric and semantic decouplers, and a Uni-Perceiver fusion module.
- Overview: The Uni-Perceiver fuses the foresight representations with original diffusion features to condition the downstream diffusion policy.This fusion supplies the action policy with both distilled foresight and original diffusion information.
- Video Diffusion Backbone: Stable Video Diffusion serves as the video generation backbone for the framework.The backbone provides the diffusion features and generated-video trajectory used elsewhere in the method.
sqrt { \a lp ha
S-VAM distills multi-step video-generation priors into coherent geometric and semantic foresight from one-step diffusion features, then uses the resulting context to guide action prediction. Its decouplers combine noisy features with current-observation representations, while self-distillation trains them against VFM targets extracted from generated videos.
- One-step features: One-step diffusion features are formed by interpolating multi-resolution up-sampling features and concatenating them along the channel dimension.The resulting feature volume aggregates the first denoising step’s internal representations at a unified spatial resolution.
- Geometric and semantic decouplers: Geometric and semantic decouplers map entangled one-step features into DPAv3 and DINOv2 representation spaces, respectively.DPAv3 supplies dynamic geometric structure, while DINOv2 supplies patch-level semantic distinctiveness.
- Geometric and semantic decouplers: Each decoupler concatenates temporally replicated current-observation features with one-step features, then applies compact projection and factorized spatiotemporal transformer blocks.Spatial and temporal aggregation are applied sequentially before projecting outputs back to the target VFM dimension.
- Self-distillation: VFM targets are extracted from the diffusion model’s own multi-step generated video and used to supervise geometric and semantic decouplers.The targets are produced by a VFM encoder after spatial interpolation to a shared resolution.
- Self-distillation: Minimizing geometric and semantic distillation losses condenses multi-step generation into a single-step shortcut with high-fidelity foresight and lower denoising latency.The losses measure squared error between each decoupler output and its corresponding VFM target.
- Action expert: The action expert concatenates geometric foresight, semantic foresight, and original diffusion features, compresses them into tokens, and conditions a diffusion policy.Cross-attention aggregates salient context into latent queries, self-attention models token interactions, and the policy predicts noise for action reconstruction.
4 Experiment
S-VAM is evaluated across simulated and real-world manipulation settings, including qualitative comparisons, benchmark results, ablations, alternative VFM targets, and real-time deployment. It achieves strong performance by combining geometric and semantic foresight with efficient control.
- Qualitative comparisons: Qualitative CALVIN and MetaWorld comparisons show S-VAM producing coherent attention trajectories aligned with instructions, unlike VPP’s erratic or diverging trajectories.
- Simulated benchmarks: S-VAM achieves a state-of-the-art average length of 4.16 on CALVIN, outperforming VPP by 0.58.
- Simulated benchmarks: 72.8% overall success on MetaWorld is the highest reported result, with S-VAM outperforming HiF-VLA by 11.9% and 15.1% on the reported comparisons.
- Ablation study: Removing geometric or semantic distillation lowers CALVIN average length from 4.16 to 4.01 or 3.99, while semantic removal reduces 5th-task success from 68.9% to 64.1%.
- Ablation study: Self-distillation, Uni-Perceiver, and original diffusion features are each important: removing them reduces average length to 3.82, 3.72, and 3.93, respectively.
- Alternative VFM representations: DINOv2 and DPAv3 provide the strongest VFM combination, while dense semantics and dynamic geometric representations outperform global semantics and static geometry.
- Real-world experiments: In real-world experiments, S-VAM improves success rates over VPP across all tasks without compromising real-time control, reaching an effective control frequency of 25 Hz.
5 Conclusion
S-VAM resolves the video-action paradigm’s efficiency–foresight dilemma by distilling multi-step visual planning into coherent geometric and semantic foresight from one-step features. Experiments support efficient and precise manipulation in both simulated and real-world settings.
- S-VAM establishes a direct shortcut between inference efficiency and foresight fidelity in video-action models.
- A self-distillation strategy maps noisy one-step diffusion features to coherent geometric and semantic foresight using visual foundation representations from multi-step generated videos.
- The shortcut preserves high-fidelity planning while maintaining single-step inference efficiency for the downstream policy.
- S-VAM achieves state-of-the-art performance on simulated benchmarks and supports efficient, precise manipulation in complex real-world scenarios.