SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy

• Existing VLA and VAM policies commonly lack a unified understanding of the environment’s 3D spatio-temporal state.Source: §1 · pp. 1–2
• SpatialVAM represents 3D structure through spatial-aware multi-view projections and target positions as multi-view heatmaps.Colored point clouds become multi-view RGB images, while target positions use the same projection mechanism.Source: §1 · p. 2
• A unified video foundation model jointly predicts future RGB frames and heatmap sequences, which are decoded into continuous action chunks.Source: §1 · p. 2, Figure 1 · p. 2
• SpatialVAM consistently outperforms VAMs, VLA models, and 3D-based policies across Meta-World, RoboCasa, and real-world benchmarks.The method is also reported as data-efficient, robust, generalizable, and interpretable, including learning from as few as 10 demonstrations without pretraining.Source: §1 · p. 2
• The paper contributes a 3D World-Action Model framework, the SpatialVAM method, and extensive simulation and real-world evaluations.Source: §1 · p. 2

2.1 Vision Language Action Models

• Vision-language action models typically use 2D observations and large vision-language backbones, but often require large robot datasets and explicitly model environment dynamics only weakly.Source: §2.1 · pp. 2–3

2.2 Video Action Models

• Video action models either separate future-video prediction from action generation or jointly predict videos and actions in one framework.Existing intermediate representations include RGB videos, human demonstrations, optical flow, and 2D point trajectories.Source: §2.2 · p. 3

2.3 3D Structural Priors for Manipulation

• 3D-aware manipulation policies use point clouds, 3D foundation models, or multi-view images with known camera transformations as structural priors.Source: §2.3 · p. 3
• Unlike prior approaches, SpatialVAM uses a video foundation model to capture spatial structure and temporal dynamics jointly.Source: §2.3 · p. 3, §2.2 · p. 3
• SpatialVAM predicts continuous actions and future observations rather than only key poses or waypoints, supporting more general manipulation skills.Source: §2.3 · p. 3

3.1 Multi-View Projection

• SpatialVAM projects cropped point clouds and end-effector poses into three fixed virtual views, producing RGB images and heatmaps for spatially aware conditioning.Orthographic projection is used, and heatmaps encode end-effector pixel locations for processing by a pretrained VAE.Source: §3.1 · pp. 3–4

3.3 Action Decoding

• Final action chunks combine predicted 3D positions, rotations, and gripper states for robot execution.A lightweight action decoder predicts rotation and gripper states from denoised video latents.Source: Figure 2 · p. 4, §3.4 · p. 6

3.2 Multi-View Video Diffusion

• The video diffusion transformer jointly predicts future multi-view RGB and heatmap videos using a pretrained backbone augmented with view-attention for cross-view interactions.Latents are concatenated across views, reshaped for per-timestep cross-view attention, and denoised before VAE decoding.Source: §3.2 · pp. 4–5

3.3 Action Decoding

• Predicted heatmap peaks are back-projected through known camera parameters to recover a continuous 3D end-effector trajectory.Denoised latents also provide global and heatmap-centered local features for predicting rotation and gripper changes.Source: §3.3 · p. 5

3.4 Training & Inference

• Training applies SE(3) augmentation, shared VAE encoding, diffusion noise prediction, and separate video, heatmap, rotation, and gripper objectives.LoRA fine-tuning reduces computational cost, while the rotation predictor uses discretized Euler angles and binary gripper states.Source: §3.4 · p. 5
• Table 1 reports seven Meta-World tasks evaluated with five demonstrations per task, with SpatialVAM achieving the highest average success rate over baselines.Each table entry counts successful rollouts out of 25 trials.Source: Table 1 · p. 6

4.1 Meta-World Experiments

• 89.1% average success with five demonstrations per task surpasses all Meta-World baselines, with SpatialVAM leading on five of seven tasks.Video-prediction methods outperform behavioral cloning, while SpatialVAM exceeds prior video-prediction approaches.Source: §4.1 · p. 6

4.2 RoboCasa Experiments

• SpatialVAM substantially outperforms both RoboCasa baselines using only 10 demonstrations and 50 evaluation rollouts per task.The comparison includes Cosmos Policy and 3D Diffuser Actor, representing video-action and 3D-based approaches.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Real-World Experiments

• With about 10 expert trajectories per task, SpatialVAM solves all three basic real-world tasks and achieves the best performance on unseen Push-L and Scoop-C settings.It slightly underperforms BridgeVLA on Put-B and Put-H, which the authors attribute to BridgeVLA’s additional image–text pretraining.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Robustness Analysis

• With one camera, adapting virtual views recovers success from 50.0% to 60.0%, while 20 mm and 2° calibration perturbations retain 51.4% success without retraining.The results indicate that view alignment and spatially smooth heatmaps reduce sensitivity to incomplete coverage and calibration errors.Source: §4.4 · pp. 8–9
• Varying the RGB loss weight by 80% changes success 3.3%, varying heatmap spread by 133% changes it 2.5%, and one denoising step remains effective.The authors recommend five denoising steps for 5 Hz inference on a single NVIDIA A100 GPU server.Source: §4.4 · p. 9, Figure 5 · p. 9

4.5 Ablation Studies

• Ablations show 87.4% with full fine-tuning, 81.1% with channel concatenation, 76.6% with latent regression, and 4.6% without pretrained video weights.These results support LoRA, view-dimension concatenation, heatmap-peak back-projection, and video-foundation-model initialization.Source: §4.5 · pp. 9–10, Table 3 · p. 10
• Success rises monotonically from 61.1% to 89.1% as additionally predicted RGB views increase from zero to three.The ablation supports jointly predicting RGB and heatmap videos and using more views for spatio-temporal understanding and control.Source: Figure 5 · p. 9, §4.5 · p. 10

4.6 Safer Deployment and Enhanced Explainability

• Video inspection before execution significantly reduces collisions across 140 user-study rollouts, providing an interpretable safety check.Predicted multi-view videos and heatmaps allow evaluators to identify potentially unsafe action sequences before physical execution.Source: §4.6 · p. 10, Figure 6 · p. 10

5 Conclusion and Future Work

• A fixed RGB-D camera and approximately known calibration constrain deployment, while 24-frame action chunks require about 4.6 s to generate on an NVIDIA A100.The authors identify diffusion acceleration as future work for real-time control.Source: §5 · p. 10

A Multi-View Video Diffusion

• SpatialVAM uses a multi-view video diffusion pipeline with a 3D VAE encoder, Diffusion Transformer, and 3D VAE decoder.The encoder processes RGB and heatmap sequences; the DiT models temporal evolution; the decoder reconstructs RGB or heatmap sequences.Source: A Multi-View Video Diffusion · p. 15, §A.2 · p. 15, §A.3 · p. 16

A.1 3D VAE Encoder

• The 3D VAE compresses input videos spatially from H × W to H/8 × W/8 and temporally from 1 + T frames to [1 + T/4].The compressed representation uses C = 16 channels and reduces the computational cost of long sequences.Source: §A.1 · p. 15

A.2 Diffusion Transformer (DiT)

• The DiT adds noise to concatenated video and heatmap latents and learns flow-matching velocity predictions to generate temporally consistent future sequences.During inference, timestep-wise velocity predictions denoise the latent sequence.Source: §A.2 · p. 15

A.4 Efficient Inference with Cache Mechanism

• Feature caching reuses previously computed features to reduce memory and computation for long video-sequence inference.The cache mechanism is intended to enable handling of arbitrarily long video sequences.Source: §A.4 · p. 16

B.1 Why Virtual-View Rendering?

• Virtual-view rendering exposes task-relevant workspace regions and preserves complementary multi-view representations even with fewer physical cameras.The observed geometry cannot recover unseen surfaces, but adapted virtual views can achieve performance comparable to the original three-camera setting.Source: §B.1 · p. 16

B.2 Projection Process

• Point-cloud projection computes depth and pixel coordinates, resolves collisions by nearest-depth Z-ordering, and applies screen-space splatting to reduce visual noise.The projection maps 3D points and RGB features into RGB and depth images, while splatting fills sparse screen-space regions.Source: §B.2 · pp. 16–17

B.3 Back Projection Process

• Back-projection averages probabilities across three heatmaps for discretized workspace points and selects the highest-probability point as the target 3D position.Applying this operation over predicted timesteps recovers a continuous end-effector trajectory.Source: §B.3 · p. 17

C Training & Inference Details

• The supplied training-and-inference section points to Table 4 for hyperparameters and Table 5 for robustness results over seven tasks and ten trials per task.The passage identifies these resources but does not provide the table values.Source: C Training & Inference Details · p. 17, Table 5 · p. 18
• Meta-World training uses 32 NVIDIA H20 GPUs for 21.8k steps, while real-world training uses 32 NVIDIA H200 GPUs for 8.6k steps.The reported training times are approximately 42 and 36 hours for Meta-World models, and 11 and 10 hours for real-world models.Source: C Training & Inference Details · p. 18
• Real-world inference requires more than 30 GB of GPU memory and averages 4.6 seconds to produce a 24-step action chunk from point-cloud input.Inference runs on an NVIDIA A100 server with FastAPI communication to the robot client.Source: C Training & Inference Details · p. 18

D Deployment Robustness Analysis

• The study separately evaluates robustness to reduced camera coverage and calibration errors across seven real-world manipulation tasks.These tests follow the real-world evaluation protocol described in Section 4.3.Source: D Deployment Robustness Analysis · p. 18

D.1 Robustness to Camera Number

• The study compares three-camera input, original virtual views with one camera, and adapted virtual views using the same seven tasks and ten trials per task.Single-camera input substantially reduces point-cloud coverage, while adapted views are selected for the remaining camera’s observable workspace.Source: §D.1 · p. 18

• Camera extrinsics are perturbed by 20 mm in translation and 2° in rotation, creating train–test mismatch or consistent calibration bias.Both settings are compared with the original calibration using 10 trials per task.Source: §D.2 · p. 18

E Robustness Analysis for Key Parameters

• 80% variation in RGB loss weight λ changes success rate by 2.9%, while 133% variation in heatmap standard deviation σ changes it by 2.2%.Each hyperparameter is varied independently while other settings remain fixed and the model is retrained and reevaluated on Meta-World.Source: E Robustness Analysis for Key Parameters · p. 18, Table 7 · p. 20
• Projected observations compare three physical cameras, one camera with original views, and one camera with adapted views for workspace coverage.A single camera reduces point-cloud coverage, whereas adapted virtual views better expose the observable workspace.Source: Figure 7 · p. 19

F Meta-World Simulation Baselines

• Meta-World comparisons include behavioral cloning, iterative action diffusion, two-stage video prediction, object-centric tracking, and a large video-action model.The baselines span BC-Scratch, BC-R3M, Diffusion Policy, UniPi, AVDC, Track2Act, and DreamZero.Source: F Meta-World Simulation Baselines · pp. 19–20

G Real-World Baselines

• Real-world comparisons cover direct 3D-to-action mapping, video-prediction policies, VLAs, and multi-view heatmap-based keyframe estimation.The evaluated baselines are DP3, π0.5, UVA, and BridgeVLA.Source: G Real-World Baselines · pp. 20–21
• DP3 uses sparse depth-reconstructed point clouds, while π0.5 processes visual observations, language, and actions with a pretrained vision-language backbone.DP3 conditions diffusion actions on encoded 3D features and proprioception; π0.5 uses hybrid tokenized and continuous action representations.Source: G Real-World Baselines · pp. 20–21
• UVA jointly predicts future videos and actions from single-view inputs, whereas BridgeVLA predicts multi-view heatmaps for a single next keyframe.BridgeVLA uses point clouds projected into orthographic views but requires an external motion planner for execution between key poses.Source: G Real-World Baselines · p. 21

• DP3 overfits severely: despite training loss falling from 10^-1 to 10^-8, it fails to approach the Put Lion target under real-world variability.The analysis attributes this failure to its simple MLP-based point-cloud encoder struggling with sensor noise and variability.Source: H Detailed Real-World Analysis · p. 21
• π0.5 shows limited spatial generalization, fails on Push T, and cannot reliably place the tortilla on the plate despite scooping it.Its Put Lion success is restricted to the training distribution, while plate-placement variation causes failures on Scoop Tortilla.Source: H Detailed Real-World Analysis · p. 21
• UVA produces reasonable action tendencies but lacks execution accuracy, causing premature closing, over-pushing, and inaccurate pouring across tasks.Its failures include premature gripper closure on Put Lion, out-of-distribution states on Push T, and tortillas falling outside the plate on Scoop Tortilla.Source: H Detailed Real-World Analysis · p. 21
• BridgeVLA is the strongest competing baseline but often fails contact-rich Scoop Tortilla because it predicts only a single key pose rather than continuous intermediate actions.Its static image–text backbone cannot predict continuous video or heatmap sequences, and an external planner handles motion between poses.Source: H Detailed Real-World Analysis · pp. 21–22
• SpatialVAM’s primary failure is inaccurate action decoding: on Scoop Tortilla, it can scrape the tortilla instead of lifting it from underneath.The authors attribute this to 256 × 256 heatmaps, where each pixel represents approximately 4 mm, and suggest higher resolution as a possible improvement.Source: H Detailed Real-World Analysis · p. 22

• At one denoising step, RGB video quality visibly deteriorates but predicted heatmap peaks remain relatively stable.Increasing denoising steps to 5 or 50 produces no significant visual difference in the generated outputs.Source: I Visualization of Video Prediction · p. 22, Figure 13 · p. 28, Figure 14 · p. 29
• RGB–heatmap consistency enables visual validation of predicted actions, providing a tool for avoiding unsafe deployments.The generated videos align well with the heatmaps in both Meta-World and real-world environments.Source: I Visualization of Video Prediction · p. 22
• Predicted RGB sequences are visually realistic, while heatmap peaks closely follow the end effector across Button-Press-Top, Door-Open, Push-T, Scoop Tortilla, and Put Lion.The visualizations overlay predicted and ground-truth heatmap peaks on their corresponding RGB images.Source: Figure 8 · p. 23, Figure 9 · p. 24, Figure 10 · p. 25