S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight

Motivation

• VLA models lack spatiotemporal foresight and depend on costly robot action data because their underlying vision-language models are trained mainly on static image-text pairs.VAMs instead exploit predictive priors from abundant internet-scale videos to reduce reliance on robot demonstrations.Source: §1 · pp. 1–2
• Multi-step video generation provides detailed planning priors but is too slow for high-frequency control, while one-step features are fast but noisy, entangled, and weak in geometry and semantics.The missing cues include geometry for monocular-depth ambiguity and semantic distinctiveness for separating task-relevant objects.Source: §1 · pp. 2–3, Figure 1 · p. 2

S-VAM

• S-VAM foresees coherent geometric and semantic representations in one forward pass to guide precise action generation.Lightweight decouplers explicitly disentangle noisy one-step diffusion features into specialized representations.Source: §1 · p. 3
• Self-distillation trains decouplers as students using DPAv3 geometry and DINOv2 semantic representations extracted from the diffusion model’s own multi-step generated videos.This condenses structured multi-step denoising priors into one-step inference.Source: §1 · p. 3

Results

• S-VAM outperforms state-of-the-art methods, enabling efficient and precise manipulation in complex scenarios.The paper reports this outcome across its evaluated settings.Source: §1 · p. 3

Vision-Language-Action Models

• VLA methods map text instructions and visual observations to actions but struggle with spatiotemporal foresight because their pretrained vision-language models primarily use static image-text data.This motivates methods that incorporate video-based predictive representations.Source: §2.1 · p. 3

Video-Action Models

• Video-action models pair video diffusion models that synthesize future visual plans with action experts that predict control signals.The paradigm leverages visual foresight from video generation for robotic action learning.Source: §2.2 · p. 4
• Multi-step video generation improves visual foresight but incurs prohibitive inference latency, motivating methods that directly extract one-step denoising features.This establishes the efficiency-fidelity trade-off addressed by S-VAM.Source: §2.2 · p. 4

Vision Foundation Models

• Vision foundation models provide fine-grained semantic context, while related approaches add explicit depth or point-cloud cues for geometric understanding.S-VAM differs by distilling geometric and semantic foresight rather than explicitly injecting these cues at the input.Source: §2.3 · p. 4

Overview

• S-VAM foresees geometric and semantic representations for action generation without multi-step denoising.The method is organized around preliminaries, geometric and semantic decouplers, and a Uni-Perceiver fusion module.Source: §3 · p. 4
• The Uni-Perceiver fuses the foresight representations with original diffusion features to condition the downstream diffusion policy.This fusion supplies the action policy with both distilled foresight and original diffusion information.Source: §3 · p. 4

Video Diffusion Backbone

• Stable Video Diffusion serves as the video generation backbone for the framework.The backbone provides the diffusion features and generated-video trajectory used elsewhere in the method.Source: §3.1 · p. 4

One-step features

• One-step diffusion features are formed by interpolating multi-resolution up-sampling features and concatenating them along the channel dimension.The resulting feature volume aggregates the first denoising step’s internal representations at a unified spatial resolution.Source: §3.2 · p. 6

Geometric and semantic decouplers

• Geometric and semantic decouplers map entangled one-step features into DPAv3 and DINOv2 representation spaces, respectively.DPAv3 supplies dynamic geometric structure, while DINOv2 supplies patch-level semantic distinctiveness.Source: §3.2 · p. 6
• Each decoupler concatenates temporally replicated current-observation features with one-step features, then applies compact projection and factorized spatiotemporal transformer blocks.Spatial and temporal aggregation are applied sequentially before projecting outputs back to the target VFM dimension.Source: §3.2 · pp. 6–7

Self-distillation

• VFM targets are extracted from the diffusion model’s own multi-step generated video and used to supervise geometric and semantic decouplers.The targets are produced by a VFM encoder after spatial interpolation to a shared resolution.Source: sqrt { \a lp ha · p. 5, §3.2 · p. 7
• Minimizing geometric and semantic distillation losses condenses multi-step generation into a single-step shortcut with high-fidelity foresight and lower denoising latency.The losses measure squared error between each decoupler output and its corresponding VFM target.Source: §3.2 · p. 7

Action expert

• The action expert concatenates geometric foresight, semantic foresight, and original diffusion features, compresses them into tokens, and conditions a diffusion policy.Cross-attention aggregates salient context into latent queries, self-attention models token interactions, and the policy predicts noise for action reconstruction.Source: §3.3 · pp. 7–8

Qualitative comparisons

• Qualitative CALVIN and MetaWorld comparisons show S-VAM producing coherent attention trajectories aligned with instructions, unlike VPP’s erratic or diverging trajectories.Source: Figure 3 · p. 9, §4.2 · pp. 9–10, Figure 4 · p. 11

Simulated benchmarks

• S-VAM achieves a state-of-the-art average length of 4.16 on CALVIN, outperforming VPP by 0.58.Source: §4.2 · p. 9
• 72.8% overall success on MetaWorld is the highest reported result, with S-VAM outperforming HiF-VLA by 11.9% and 15.1% on the reported comparisons.Source: §4.2 · pp. 10–11

Ablation study

• Removing geometric or semantic distillation lowers CALVIN average length from 4.16 to 4.01 or 3.99, while semantic removal reduces 5th-task success from 68.9% to 64.1%.Source: §4.3 · p. 12
• Self-distillation, Uni-Perceiver, and original diffusion features are each important: removing them reduces average length to 3.82, 3.72, and 3.93, respectively.Source: §4.3 · p. 12

Alternative VFM representations

• DINOv2 and DPAv3 provide the strongest VFM combination, while dense semantics and dynamic geometric representations outperform global semantics and static geometry.Source: §4.4 · p. 13

Real-world experiments

• In real-world experiments, S-VAM improves success rates over VPP across all tasks without compromising real-time control, reaching an effective control frequency of 25 Hz.Source: Figure 5 · p. 14, §4.5 · p. 15

• S-VAM establishes a direct shortcut between inference efficiency and foresight fidelity in video-action models.Source: §5 · p. 15
• A self-distillation strategy maps noisy one-step diffusion features to coherent geometric and semantic foresight using visual foundation representations from multi-step generated videos.Source: §5 · p. 15
• The shortcut preserves high-fidelity planning while maintaining single-step inference efficiency for the downstream policy.Source: §5 · p. 15
• S-VAM achieves state-of-the-art performance on simulated benchmarks and supports efficient, precise manipulation in complex real-world scenarios.Source: §5 · p. 15