Source-linked AI summary
VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
Yihao Wang, Pengxiang Ding, Lingxiao Li, Can Cui, Zirui Ge, Xinyang Tong, Wenxuan Song, Han Zhao, Wei Zhao, Pengxu Hou, Siteng Huang, Yifan Tang, Wenhui Wang, Ru Zhang, Jianyi Liu, Donglin Wang
TL;DR
VLA models face costly dependence on large VLMs and robotic-data pre-training, motivating more effective VL-to-action bridging. VLA-Adapter analyzes VL conditions and introduces Bridge Attention in a lightweight Policy, achieving strong performance with a tiny backbone while lowering deployment barriers.
Problem
VLA models rely on large-scale VLMs and extensive robotic-data pre-training, motivating the question of how to bridge VL representations to actions more effectively.
Method
VLA-Adapter systematically analyzes VL conditions and uses Bridge Attention in a lightweight Policy to inject the optimal condition into the action space.
Results
VLA-Adapter achieves SOTA-level performance with a tiny-scale backbone, including strong results with frozen VLMs, low VRAM usage, and high inference speed.
Takeaways & Limitations
VLA-Adapter alleviates reliance on large-scale VLMs and huge training costs, lowering the barrier to deploying VLA models.
Takeaways & Limitations
Because it lacks large-scale embodied-data pre-training and uses a tiny scale, VLA-Adapter's real-world generalization needs improvement.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models typically bridge the gap between perceptual and action spaces by pre-training a large-scale Vision-Language Model (VLM) on robotic data. While this approach greatly enhances performance, it also incurs significant training costs. In this paper, we investigate how to effectively bridge vision-language (VL) representations to action (A). We introduce VLA-Adapter, a novel paradigm designed to reduce the reliance of VLA models on large-scale VLMs and extensive pre-training. To this end, we first systematically analyze the effectiveness of various VL conditions and present key findings on which conditions are essential for bridging perception and action spaces. Based on these insights, we propose a lightweight Policy module with Bridge Attention, which autonomously injects the optimal condition into the action space. In this way, our method achieves high performance using only a 0.5B-parameter backbone, without any robotic data pre-training. Extensive experiments on both simulated and real-world robotic benchmarks demonstrate that VLA-Adapter not only achieves state-of-the-art level performance, but also offers the fast inference speed reported to date. Furthermore, thanks to the proposed advanced bridging paradigm, VLA-Adapter enables the training of a powerful VLA model in just 8 hours on a single consumer-grade GPU, greatly lowering the barrier to deploying the VLA model. Project page: https://vla-adapter.github.io/.
1 INTRODUCTION
VLA models commonly depend on large-scale VLMs and embodied-data pre-training, creating efficiency bottlenecks. VLA-Adapter addresses the VL-to-action bridge with a lightweight policy and reports strong performance at lower deployment cost.
- VLA models often rely on large-scale embodied data to pre-train VLMs before a Policy network generates actions.
- High-dimensional control exposes bottlenecks in VLA models, including large VLM dependence, slow fine-tuning, high VRAM use, and low inference throughput.
- VLA-Adapter systematically analyzes how VL conditions affect action generation and uses the findings to design its bridging paradigm.
- The method transfers sufficient multimodal information to a Policy network, effectively bridging vision-language representations and actions.
- Rich experiments report higher success rates, smaller scale, lower tuning cost, and faster inference across simulated and real-world robotic tasks.
2 RELATED WORK
Related VLA work has evolved from direct action-token alignment toward continuous actions and multiple VL-to-action bridging interfaces. The effectiveness of these paradigms remains difficult to compare systematically.
- VLA systems typically combine large-scale embodied-data pre-training with a Policy network that generates action sequences for diverse tasks.
- The central design challenge is bridging vision-language perception space to action space effectively and efficiently.
- Earlier methods discretized actions into tokens, but this approach inevitably introduces inherent loss; newer studies emphasize continuous action spaces.
- Raw Features from VLMs: Raw-feature bridges use VLM representations from final, middle, first-half, or all intermediate layers to provide semantic or multimodal information.
- Additional Query as Interface: Additional-query interfaces use learnable queries as bridges between VLMs and Policy instead of transmitting raw features.
3 VLA-ADAPTER METHODOLOGY
VLA-Adapter bridges vision-language representations to actions by exploring effective conditions and using a lightweight Policy with Bridge Attention. Its findings favor middle-layer Raw features, deep-layer ActionQuery features, and multi-layer conditions, which are integrated into an end-to-end action-generation Policy.
- VLA-Adapter Framework: The VLM processes third-view and gripper images, language, and ActionQuery inputs to produce Raw and ActionQuery latents that condition the Policy.Vision embeddings are extracted with DINOv2 and SigLIP, while the instruction is tokenized.
- Effective Condition Exploration: The study compares VLM feature layers and feature types to determine which perception information is effective for action generation.It asks whether layer choice matters and whether ActionQuery features outperform Raw features.
- Key Findings: Middle-layer CR_t performs better than deep-layer CR_t because it retains richer multimodal details for action generation.Deep-layer CR_t is described as biased toward semantic information, whereas middle-layer CR_t integrates image and text information.
- Key Findings: Deep-layer CAQ_t performs better than other-layer CAQ_t because it aggregates richer multimodal details and promotes action generation.ActionQuery features are trained from scratch in the explored framework.
- Key Findings: All-layer features generally outperform single-layer features and avoid selecting the best layer during design.The paper presents this as a more universal design choice.
- Policy with Bridge Attention: Bridge Attention combines two cross-attention operations with one self-attention operation to inject Raw and ActionQuery conditions into action latents.The Policy processes corresponding-layer action latents, proprioceptive state, and initial action latents, then yields an H-step action chunk.
4 EXPERIMENTS
Experiments evaluate VLA-Adapter’s bridge paradigm across simulated and real-world robotic benchmarks, comparing backbone choices, frozen-backbone settings, efficiency, generalization, and ablations. Results indicate strong performance with a tiny backbone, faster inference, and benefits from multimodal condition design.
- 4.1 NECESSITY OF VLA-ADAPTER: Experiments compare VLA-Adapter with OpenVLA-OFT using backbones ranging from 0.5B to 7B parameters, including a robotic-data-pretrained OpenVLA-7B.The comparison uses LIBERO-Long and includes both non-pretrained and robotic-data-pretrained backbones.
- 4.1 NECESSITY OF VLA-ADAPTER: VLA-Adapter remains effective when the backbone is frozen, with only ActionQuery and Policy trained from scratch.The frozen-backbone comparison includes OpenVLA-OFT and SmolVLA.
- 4.1 NECESSITY OF VLA-ADAPTER: VLA-Adapter shows clearer improvement for VLMs without robotic pre-training, while remaining strong when the backbone is frozen.The authors attribute this contrast to whether last-layer features are already adapted to the action domain.
- 4.1 NECESSITY OF VLA-ADAPTER: VLA-Adapter attains faster inference speed than the compared OpenVLA systems.Inference efficiency is evaluated with an 8-dimensional action chunk against OpenVLA and OpenVLA-OFT.
- 4.2 OVERALL PERFORMANCE ON VARIOUS TASKS: On LIBERO, VLA-Adapter performs comparably to OpenVLA-OFT with 14× larger scale, surpasses π0, SmolVLA, and GR00T N1, and gains 29.0% over VLA-OS on LIBERO-Long.Success Rate is evaluated over repeated subtasks, with higher values indicating better performance.
- 4.3 PERFORMANCE ON GENERALIZATION TASKS: VLA-Adapter shows strong generalization on CALVIN ABC→D and better average length than SOTA baselines, while real-world experiments report better generalization across scenarios.The real-world results average 10 executions.
- 4.5 ABLATION EXPERIMENTS: Using too few ActionQuery tokens weakens multimodal aggregation, whereas too many introduce redundancy that interferes with performance.The study varies ActionQuery counts from 1 to 512.
- 4.5 ABLATION EXPERIMENTS: Using all-layer Raw and ActionQuery conditions achieves superior performance, and Bridge Attention benefits from fully injecting aggregated ActionQuery information into Policy.The ablations compare condition types and injection degrees.
5 CONCLUSION
The conclusion presents VLA-Adapter as an efficient VLA bridging paradigm that transfers multimodal information to Policy. The reported results combine strong performance, frozen-backbone effectiveness, low VRAM use, and high inference speed to lower deployment barriers.
- 5 CONCLUSION: VLA-Adapter transfers Raw and ActionQuery latent information to Policy for action generation.The method is described as a bridging paradigm for transferring multimodal knowledge from VL to A.
- 5 CONCLUSION: VLA-Adapter achieves SOTA performance with a tiny-scale backbone and remains strong when the VLM is frozen.The conclusion also reports low VRAM usage and high inference speed.
- 5 CONCLUSION: The reported efficiency and performance suggest reduced reliance on large-scale VLMs and high training costs, lowering the barrier to VLA deployment.This is the paper’s stated practical implication.
6 LIMITATIONS
The paper identifies limitations concerning real-world generalization, dependence on VLM-provided conditions, and the simplicity of the current training process.
- 6 LIMITATIONS: Because VLA-Adapter lacks large-scale embodied-data pre-training and uses a tiny scale, its real-world generalization still needs improvement.The limitation is stated specifically for real-world systems.
- 6 LIMITATIONS: Policy action quality depends on the conditions provided by the VLM and how those conditions are used.The authors propose further exploration of these conditions.
- 6 LIMITATIONS: The current training process remains relatively simple, leaving complex procedures such as reinforcement learning for future exploration.This is presented as a future-work direction.
A SETUP DETAILS OF LIBERO SIMULATION BENCHMARKS
LIBERO evaluates instruction-conditioned robot control across four task suites, using visual inputs and repeated trials to measure subtask success.
- Benchmark composition: LIBERO contains four suites: Spatial, Object, Goal, and LIBERO-100.The first three suites contain 10 tasks each; LIBERO-100 contains 90 short-term and 10 long-horizon tasks.
- Evaluation protocol: Each task is repeated 50 times to calculate the average success rate for each subtask.
- Task instructions: The benchmark includes spatial, goal-oriented, object-placement, and long-horizon instruction examples.Examples include placing objects, opening drawers, operating a stove, and arranging tableware.
- Task instructions: LIBERO-100 includes long-horizon tasks represented by multi-step instructions.The supplied examples list ten long-task instructions, including compound object-placement sequences.
- Inputs and control: The policy receives third-person and wrist images, plus an instruction prompt, and outputs a 7-dimensional action vector for a simulated 7-DOF Panda arm.Both image inputs have resolution 224×224×3 and RGB channels.
B.1 OVERALL ARCHITECTURE
The overall architecture uses a DiT-based Policy aligned in depth with the VLM, repeatedly integrating multimodal conditions through Bridge Attention to produce action chunks.
- Policy architecture: The Policy contains τ DiT blocks, with the number of layers matched to the VLM.Each block combines conditional modulation, conditional attention, and a conditional feedforward network.
- Conditional modulation: AdaLN-Zero modulates intermediate activations so noisy action sequences can transition toward fine-grained predictions.It uses scaling, shifting, and gated residual control to adjust activation amplitude.
- Bridge Attention: Bridge Attention uses processed action latents with condition vectors to inject multimodal information into the action representation.The resulting attention latent is passed to the conditional feedforward network.
- Action generation: After M DiT blocks, LayerNorm and an MLP transform the final latent into the current action chunk.
- Training: The DiT-based Policy is trained from scratch by adding Gaussian noise to ground-truth action trajectories.The noisy trajectory follows the stated diffusion formulation and is optimized with the Policy training objective.
- Policy comparison: On LIBERO-Long, the L1-based Policy outperforms the DiT-based Policy and achieves higher throughput.The reported pattern is consistent with OpenVLA-OFT’s comparison between pre-training and fine-tuning.
C DETAILED COMPARISON RESULTS OF DIFFERENT CODITIONS
Condition comparisons on LIBERO-Long identify layer-dependent preferences for Raw and ActionQuery features, motivating the use of both feature types across all layers.
- Raw features: Middle-layer Raw features generally outperform Raw features from other layers.
- ActionQuery features: Deep-layer ActionQuery features generally outperform shallow-layer ActionQuery features.
- Condition selection: Using features from all layers gives the best performance across the reported Raw and ActionQuery comparisons.VLA-Adapter therefore uses all-layer Raw and ActionQuery features as Policy conditions.
- Benchmark results: VLA-Adapter performance is reported across 40 subtasks spanning the four LIBERO suites.The detailed results are provided in Table D1.
E SETUP DETAILS OF CALVIN SIMULATION BENCHMARK
CALVIN ABC→D evaluates zero-shot generalization by training on three environments and testing on a held-out fourth environment using multimodal inputs for simulated robot control.
- Benchmark protocol: CALVIN ABC→D trains on Env A, B, and C, then evaluates on Env D.CALVIN contains four environments and 34 different subtasks.
- Task protocol: The benchmark uses a preset sequence of 1,000 tasks, with each task row containing five subtasks.The model proceeds to the next subtask only after completing the current one.
- Inputs and control: CALVIN provides third-person and gripper images together with task instructions to control a simulated 7-DOF Franka Emika Panda arm.Third-person images are 224×224×3 RGB, while gripper images are 84×84×3 RGB.
F.1 TRAINING DETAILS
The training details specify the optimizer, learning-rate schedule, and batch size, while the accompanying figures document execution examples across real-world, LIBERO, and CALVIN tasks.
- Training configuration: VLA-Adapter training uses AdamW with LoRA, a learning rate of 1e-4, cosine annealing with warm-up, and batch size 16.These settings are accompanied by dedicated tables listing training details and hyperparameters.
- Execution examples: Execution examples are provided for real-world tasks and LIBERO and CALVIN ABC→D tasks.The examples are referenced in Figures G1 and G2.
- Evaluation examples: The real-world evaluation includes long-horizon and short-horizon manipulation tasks.Examples include placing a spoon on a cup followed by placing the cup on a plate, stacking blocks, moving a block, and placing a duck on a plate.
H EFFECTIVENESS ANALYSIS OF FROZEN BACKBONE
The frozen-backbone analysis reports that OpenVLA-OFT does not work, while VLA-Adapter remains effective because its learnable ActionQuery is added outside the frozen VLM. The section also documents lightweight and Pro architectural variants built around attention-based conditioning.
- Frozen-backbone effectiveness: OpenVLA-OFT does not work with the frozen backbone, despite also using learnable tokens.The passage attributes its implementation to learnable tokens but does not provide a successful frozen-backbone result for that method.
- Frozen-backbone effectiveness: VLA-Adapter remains effective because ActionQuery tokens are inserted into the sequence and learned from scratch outside the original frozen VLM.This differs from supplying the VLM with an initially zero mask that cannot be trained when the backbone is frozen.
- Architecture variants: The lightweight VLA-Adapter shares one projection layer across all three attention matrices, resulting in a parameter count of 97MB.VLA-Adapter-Pro instead separates the projection layers, increasing the parameter count to 207MB and adding Rotary Position Embedding to QK.
- Bridge attention: Bridge attention combines self, adapter, and task token values, with a tanh-transformed gating factor scaling task-token attention scores.The implementation computes attention weights over concatenated token sources and projects the resulting output.
- Evaluation scope: The reported comparison between VLA-Adapter and VLA-Adapter-Pro covers 40 subtasks across four LIBERO suites.The detailed results are provided in Table I1.