PaperlayerView exact revision on arXiv

Source-linked AI summary

SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead

Chaojun Ni, Cheng Chen, Xiaofeng Wang, Zheng Zhu, Wenzhao Zheng, Boyuan Wang, Tianrun Chen, Guosheng Zhao, Haoyun Li, Zhehao Dong, Qiang Zhang, Yun Ye, Yang Wang, Guan Huang, Wenjun Mei

arXiv:2512.00903v1cs.CVcs.RO

TL;DR

Large VLA models are costly to deploy, while lightweight models compromise spatial and spatiotemporal reasoning. SwiftVLA adds cached 4D features, Fusion Tokens, and mask-and-reconstruct training to a compact VLM, matching models up to 7× larger while running 18× faster and using 12× less memory on edge devices.

  • Problem

    Lightweighting VLA models can degrade spatial reasoning and fine-grained control, while existing 3D-fusion methods rely on heavyweight VLMs or add substantial overhead.

  • Method

    SwiftVLA combines a pretrained 4D visual geometry transformer with a temporal cache, Fusion Tokens, and mask-and-reconstruct training in a compact VLM.

  • Results

    SwiftVLA matches VLA models up to 7× larger, runs 18× faster than π0, and uses 12× less memory on edge devices.

  • Takeaways & Limitations

    SwiftVLA preserves lightweight deployment efficiency while incorporating 4D spatiotemporal information and retaining comparable performance when 4D inputs are omitted at inference.

  • Takeaways & Limitations

    Lightweight VLA designs can still suffer degraded spatial reasoning and fine-grained control when backbone parameters are reduced.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models built on pretrained Vision-Language Models (VLMs) show strong potential but are limited in practicality due to their large parameter counts. To mitigate this issue, using a lightweight VLM has been explored, but it compromises spatiotemporal reasoning. Although some methods suggest that incorporating additional 3D inputs can help, they usually rely on large VLMs to fuse 3D and 2D inputs and still lack temporal understanding. Therefore, we propose SwiftVLA, an architecture that enhances a compact model with 4D understanding while preserving design efficiency. Specifically, our approach features a pretrained 4D visual geometry transformer with a temporal cache that extracts 4D features from 2D images. Then, to enhance the VLM's ability to exploit both 2D images and 4D features, we introduce Fusion Tokens, a set of learnable tokens trained with a future prediction objective to generate unified representations for action generation. Finally, we introduce a mask-and-reconstruct strategy that masks 4D inputs to the VLM and trains the VLA to reconstruct them, enabling the VLM to learn effective 4D representations and allowing the 4D branch to be dropped at inference with minimal performance loss. Experiments in real and simulated environments show that SwiftVLA outperforms lightweight baselines and rivals VLAs up to 7 times larger, achieving comparable performance on edge devices while being 18 times faster and reducing memory footprint by 12 times.

1. Introduction

SwiftVLA targets the efficiency–reasoning trade-off in lightweight VLAs by adding 4D spatiotemporal understanding to a compact VLM while preserving efficient inference. It combines 4D features, Fusion Tokens, and mask-and-reconstruct training to support action generation with minimal overhead.

  • 1. Introduction: Lightweight VLMs reduce deployment cost but weaken spatial reasoning, lowering localization, trajectory precision, and task success in complex manipulation.SmolVLA is faster than π0 but has a notably lower task success rate in the cited comparison.
  • 1. Introduction: Existing 3D-fusion methods either rely on heavyweight VLMs or add a dedicated spatial branch with substantial parameter overhead.These alternatives improve spatial awareness but remain unsuitable for compact architectures.
  • 1. Introduction: SwiftVLA extracts 4D features from images using a pretrained visual geometry transformer and temporal cache, then fuses them with 2D features through learnable Fusion Tokens.Fusion Tokens receive future end-effector trajectory supervision to produce unified, action-aware representations.
  • 1. Introduction: SwiftVLA matches VLA models up to 7× larger, runs 18× faster than π0, and uses 12× less memory on edge devices.Experiments cover simulated and real-world environments, with comparable performance to the larger baseline.
  • 1. Introduction: Mask-and-reconstruct training distills 4D knowledge into the VLA, allowing 4D inputs and auxiliary heads to be removed at inference while minimizing performance loss.During training, masked visual features are reconstructed while the model learns to generate actions.

2. Related Work

Related work reduces VLA size or adds geometric inputs, but lightweighting can impair spatial reasoning while decoupled 3D branches increase overhead. Temporal methods improve spatiotemporal perception, yet may add inference cost, motivating SwiftVLA’s lower-cost 4D design.

  • 2.1. Lightweight VLA Models: Shrinking VLA backbones reduces model size but degrades spatial reasoning and fine-grained control capabilities.This limitation affects lightweight designs derived from smaller backbones or reduced model capacity.
  • 2.2. 3D Perception in VLA Models: Directly injecting 3D features improves geometric awareness but typically depends on heavyweight VLMs for multimodal fusion.Prior methods encode spatial embeddings or geometric features into VLM representations.
  • 2.2. 3D Perception in VLA Models: Decoupled 3D designs preserve pretrained 2D representations through auxiliary or parallel branches, but increase memory footprint and inference latency.These approaches also focus on 3D information without fully addressing temporal dynamics.
  • 2.2. 3D Perception in VLA Models: SwiftVLA incrementally extracts 4D features by processing multi-view observations and reusing recent representations through a temporal cache.The cache retains only the most recent K feature representations using a FIFO strategy.
  • 2.2. 3D Perception in VLA Models: 4D-VLA adds temporal modeling through history-based keyframe sampling, but processing multiple frames introduces additional inference overhead.SwiftVLA is positioned as a lower-cost alternative for injecting 4D cues.

3. Method

SwiftVLA combines a compact VLM with incremental 4D features, Fusion Tokens, and mask-and-reconstruct training to support spatiotemporal action generation with a lightweight deployed model.

  • 3.2. Incremental 4D Feature Extraction: A pretrained 4D visual geometry transformer incrementally extracts temporally enhanced features from streaming views using spatial-temporal attention and a FIFO temporal cache.The cache retains only the most recent K 4D representations, supporting temporal context with bounded memory.
  • 3.3. Fusion Tokens: The method addresses lightweight VLMs’ difficulty fusing multimodal inputs into coherent 3D-aware representations for action generation.Fusion Tokens are introduced specifically to improve the use of 2D and 4D features within the compact VLM.
  • 3.3. Fusion Tokens: SwiftVLA fuses 2D and 4D features with Fusion Tokens supervised by future end-effector trajectories to form action-aware representations.The fused representation conditions the action expert, while trajectory supervision aligns multimodal features with spatiotemporal semantics.
  • 3.4. Mask and Reconstruct Strategy: The mask-and-reconstruct strategy randomly masks 2D or 4D features, trains action prediction alongside feature reconstruction, and distills geometry-aware representations into the VLA.The total objective combines reconstruction, action prediction, and trajectory losses.
  • 3.4. Mask and Reconstruct Strategy: At inference, SwiftVLA retains only the 2D branch, removing the 4D extractor and auxiliary heads while preserving a compact VLM-and-action-expert deployment.The 4D inputs and reconstruction components are used for auxiliary supervision during training and discarded for deployment.

4. Experiments

SwiftVLA is evaluated across simulation, real-world manipulation, and edge-device deployment against VLA models spanning multiple parameter scales. The experiments examine benchmark performance, deployment efficiency, and the contributions of 4D inputs, Fusion Tokens, reconstruction, and temporal caching.

  • Experimental Setup: The evaluation uses success rate as its primary metric and average trajectory length as an additional measure across RoboTwin, LIBERO, and real-world tasks.RoboTwin uses short-, medium-, and long-horizon tasks, while LIBERO covers four task suites.
  • Simulation Benchmark Experiments: SwiftVLA matches large VLA performance while retaining a compact design, outperforming smaller baselines on LIBERO comparisons.The comparison includes spatio-temporal enhanced, small, and large VLA categories.
  • Real-World Experiment: In real-world grasping under identical initial placements, SwiftVLA completes stable, accurate grasps while SmolVLA misses and displaces the object.The reported SmolVLA collision is associated with possible task failure or safety hazards.
  • Edge Deployment: 18× faster inference and 12× lower memory than π0 are reported on NVIDIA Jetson Orin, with latency nearly identical to SmolVLA.SwiftVLA also maintains a high success rate compared with π0 on the edge platform.
  • Ablation Study: Adding 4D features improves success rate over 2D-only inputs, while Fusion Tokens further improve the model’s use of cross-modal cues.The ablation attributes the improvement to stronger 4D representations and better alignment of 2D and 4D information.
  • Ablation Study: Mask-and-reconstruct training preserves performance comparable to full 4D inference even when 4D features are removed at inference.The study also reports that randomized temporal-cache lengths outperform the tested fixed lengths.

5. Conclusion

SwiftVLA combines cached 4D visual features, Fusion Tokens, and mask-and-reconstruct training to add spatiotemporal reasoning to a lightweight VLA. It matches models with up to 7× more parameters while achieving up to 18× faster inference and 12× lower edge-device memory use.

  • 5. Conclusion: SwiftVLA combines a temporal-cache 4D visual geometry transformer, Fusion Tokens, and mask-and-reconstruct training in a lightweight VLA.Fusion Tokens are supervised by future end-effector trajectories, while reconstruction distills 4D knowledge for inference without 4D inputs.
  • 5. Conclusion: SwiftVLA matches models with up to 7× more parameters while offering up to 18× faster inference and 12× smaller memory footprint on edge devices.These results are reported across the paper’s simulation and real-world experiments.
  • 5. Conclusion: The conclusion presents SwiftVLA as a lightweight framework for strong spatiotemporal reasoning with design efficiency.The paper includes simulation examples from four LIBERO task suites.

A. Architecture Design.

The lightweight architecture uses SmolVLM for visual perception and reduces its visual-token and layer-processing costs to accelerate inference.

  • A. Architecture Design.: SmolVLM encodes visual features with SigLIP before passing them to the language decoder.The backbone uses global image information and pixel-shuffle operations in its visual processing.
  • A. Architecture Design.: Each frame is constrained to 64 visual tokens through global image processing and pixel-shuffle operations.This reduces the number of visual tokens supplied to the model.
  • A. Architecture Design.: SwiftVLA accelerates inference by using only the first 16 layers of the VLM.The architecture omits later VLM layers during processing.

B. Implementation Details.

The implementation evaluates SwiftVLA against VLA baselines across simulated horizons and real-world manipulation tasks, using shared training procedures and task-specific benchmarks.

  • B. Implementation Details.: SwiftVLA is pretrained in two stages, beginning with action-only supervision before introducing its additional training components.The first stage runs for 100,000 steps with a global batch size of 256.
  • B. Implementation Details.: RoboTwin 2.0 organizes tasks into short, medium, and long horizons, while the real-world evaluation includes four manipulation tasks.The listed real-world tasks include Clean the Desk, Throw the Bottle, Stack Bowls, and Fold the Cloth.
  • B. Implementation Details.: The implementation reports task success rate and average trajectory length for real-world comparisons, including a separate Fold the Cloth table.The real-world task descriptions specify distinct object-manipulation requirements.

C. More Challenging Real-World Experimental Results.

SwiftVLA is evaluated on the challenging real-world Fold the Cloth task, which combines long-horizon manipulation with deformable-object dynamics. It shows strong and reliable performance compared with similarly sized models.

  • The Fold the Cloth task tests long-horizon reasoning and manipulation under complex deformable-object dynamics.
  • SwiftVLA demonstrates strong and reliable success on cloth folding, whereas SmolVLA achieves very low success rates.The comparison is performed on an AgileX PiPER six-degree-of-freedom arm with an NVIDIA RTX 4090 GPU.
  • The results highlight the advantages of incorporating 4D features for deformable-object and long-horizon manipulation tasks.

D. Supplementary Video

The supplementary video compares SwiftVLA with π0 across several real-world manipulation tasks on an NVIDIA Jetson Orin. It particularly illustrates SwiftVLA’s recovery behavior and handling of deformable objects.

  • The video compares SwiftVLA and π0 on Fold the Cloth, Throw the Bottle, and Clean the Desk using an NVIDIA Jetson Orin.
  • SwiftVLA shows superior error-correction capability over π0 when handling deformable objects in the Fold the Cloth task.
  • Compared with π0, SwiftVLA recovers more quickly with smoother trajectories for more fluid and accurate deformable-object handling.
  • The video also provides additional SwiftVLA examples on the Fold the Cloth task.
Loading 2512.00903v1…