PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

arXiv:2601.03309v2cs.CVcs.AI

TL;DR

The paper asks how VLM choice and competence translate into downstream VLA performance, a question that existing VLA designs have not systematically isolated. It introduces the lightweight VLM4VLA pipeline for fair comparison and finds that general VLM competence and auxiliary embodied skills do not reliably predict control, while the vision module is the main bottleneck. The results indicate a persistent gap between VLM pretraining and embodied action-planning requirements.

  • Problem

    Existing VLA research has given limited systematic attention to how VLM backbone choice and competence affect downstream embodied control.

  • Method

    VLM4VLA converts diverse VLMs into VLA policies with a lightweight adaptation pipeline and evaluates backbone choice, auxiliary embodied-task fine-tuning, and modality-specific training across three benchmarks.

  • Results

    VLM initialization consistently helps over training from scratch, but general VLM capability and auxiliary embodied-task gains poorly predict downstream control, while the vision encoder is the primary bottleneck.

  • Takeaways & Limitations

    Effective embodied control requires control-relevant visual representations beyond standard VLM competence and auxiliary embodied-task performance.

  • Takeaways & Limitations

    The evaluation excludes physical-robot experiments and instead uses diverse simulation benchmarks for fairness, reproducibility, and efficiency.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM's general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance. Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning.

1 INTRODUCTION

The paper introduces VLM4VLA to fairly study how VLM choice, auxiliary embodied fine-tuning, and modality-specific training affect downstream VLA performance. Across benchmarks, VLM general capability is an unreliable predictor, while vision-side alignment is especially important.

  • VLM4VLA framework: VLM4VLA converts general-purpose VLMs into VLA policies with fewer than 1% new parameters for controlled comparison.The pipeline uses a simple adaptation design intended to minimize confounding architectural variables.
  • Contribution: VLM4VLA remains competitive with more advanced action-policy designs, providing a foundation for scalable comparisons across VLM backbones.Its simple MLP-based design supports fair experiments while achieving competitive benchmark performance.
  • Study scope: The study evaluates 24 VLMs across downstream tasks and three benchmarks, examining general capability, embodied-specific capability, and modality-level contributions.The experiments compare VLM initialization, auxiliary embodied-task fine-tuning, and freezing or fine-tuning vision and language modules.
  • Main findings: VLM initialization consistently outperforms training from scratch, but standard VLM capabilities poorly predict downstream VLA performance across environments.The paper reports inconsistent backbone rankings and different correlations between general VLM evaluations and downstream benchmarks.
  • Main findings: Fine-tuning VLMs on specific auxiliary embodied tasks does not reliably improve downstream control performance.The study therefore treats embodied-task competence as insufficient evidence of effective transfer to manipulation policies.
  • Main findings: Modality ablations identify the vision encoder, rather than the language component, as the primary bottleneck for strong control performance.Injecting action-relevant information into visual modules produces significant gains and exposes a gap between VLM pretraining and embodied visual requirements.

2 RELATED WORKS

Prior VLA work has increasingly integrated VLMs into robot policies and explored embodiment-related auxiliary tasks. However, existing research has largely emphasized architecture and auxiliary-task design while giving less systematic attention to the VLM backbone itself.

  • Vision-Language-Action Models: VLA models combine pretrained VLMs with robot policies to improve generalization, using either language-token action prediction or continuous-action policy heads.The field has evolved from autoregressive action tokenization toward hierarchical VLM-plus-policy-head designs.
  • Vision-Language-Action Models: Most prior VLA methods focus on complex policy networks, while the impact of the VLM backbone itself remains comparatively underexplored.Earlier backbone comparisons also lacked consistency in implementation, motivating a fairer evaluation framework.
  • Embodied Tasks for VLM: Embodied auxiliary tasks target spatial understanding and task execution through dense prediction and autoregressive VQA objectives.These tasks are intended to improve capabilities relevant to embodied interaction and manipulation.
  • Embodied Tasks for VLM: The paper positions its benchmark as a test of whether auxiliary VLM tasks improve VLA performance rather than assuming that correspondence.The cited research direction motivates validating auxiliary-task effectiveness against VLA baselines.

3 STUDY DESIGN

The study uses a lightweight, standardized pipeline to convert diverse VLMs into comparable VLA policies and evaluates them across three simulation benchmarks. Its design emphasizes reproducibility, consistent training, and controlled modality ablations.

  • Framework principles: VLM4VLA is built around fairness, reproducibility, minimalist architecture, and isolation of intrinsic VLM capabilities.The framework uses consistent model and training settings while minimizing the influence of extraneous policy designs.
  • Study scope: The pipeline studies VLMs from 1B to 10B parameters alongside auxiliary embodied tasks and alternative VLM-to-VLA transfer settings.The design covers visual grounding, action prediction, planning, video understanding, and spatial reasoning as relevant capabilities.
  • Training: The study fine-tunes all VLM parameters with consistent configurations and evaluates rollouts using success rates across target environments.The authors report that freezing parts of the model causes significant performance degradation in their setup.
  • Network design: A learnable ActionQuery token extracts embodiment-related knowledge from the VLM, and a small MLP policy head decodes it into an action chunk.The token sequence is adapted to each VLM’s pretraining format.
  • Evaluation protocol: Evaluation uses Calvin ABC-D, SimplerEnv Bridge, and Libero-Long under standardized training and testing protocols.The benchmarks use fixed training steps and randomized evaluation trials, with task-specific success measures.

4 EXPERIMENTS AND ANALYSIS

Across three benchmarks, the experiments show that VLM choice transfers inconsistently to VLA performance, while vision-encoder adaptation is more consequential than language-embedding training. VLM pretraining remains important, but downstream control requires control-relevant visual alignment.

  • 4.1.2 MAIN RESULTS: QwenVL models significantly outperform other VLMs on Calvin ABC-D, and their stronger VQA performance coincides with superior Calvin results.The reported correlation is environment-specific rather than universal across benchmarks.
  • 4.1.2 MAIN RESULTS: KosMos-2 achieves the highest success rate on both Simpler-Bridge and Libero-10, despite being the smallest evaluated model.Paligemma outperforms Qwen2.5VL on Simpler-Bridge, whereas VLM-category performance is comparable on Libero-10.
  • 4.1.2 MAIN RESULTS: General VLM capability correlates highly with Calvin performance but shows varying or weaker relationships in Simpler and Libero environments.The analysis compares general-purpose VQA or proprietary-task scores with VLA-task performance using linear regression.
  • 4.2 IMPACT OF DIFFERENT VLM AUXILIARY TASKS ON VLA PERFORMANCE: Auxiliary embodied or generation-task fine-tuning does not reliably improve downstream control, and many fine-tuned models underperform the original baseline with higher variance.The reported examples include depth and semantic-map generation, Robobrain2, and multiple auxiliary-task settings.
  • 4.3 IMPORTANCE OF DIFFERENT VLM MODULES: Freezing the vision encoder causes significant degradation on Calvin and Simpler, whereas freezing word embeddings has no noticeable impact on VLA performance.The degradation persists despite the frozen Qwen2.5VL-7B retaining more tunable parameters than Qwen2.5VL-3B.
  • 4.4 ANALYSIS OF THE VISUAL GAP BETWEEN VLM AND VLA: Fine-tuning the vision encoder on real-world control data improves performance, supporting a semantic gap between VLM visual features and low-level manipulation requirements.The evidence argues that this effect is not primarily caused by simulation artifacts or a sim-to-real visual gap.
  • 4.4 ANALYSIS OF THE VISUAL GAP BETWEEN VLM AND VLA: VLM pretraining is crucial for VLA generalization even though pretrained visual representations remain misaligned with downstream control.Training VLAs from scratch leads to performance collapse on Calvin and Simpler, while VLM and VLA representation-learning trajectories later diverge.

5 CONCLUSIONS

The study finds a persistent mismatch between VLM capabilities and VLA embodied-task demands, while simulation-based evidence substitutes for unavailable physical-robot experiments.

  • 5 CONCLUSIONS: Figure 5 indicates that VLM and VLA learning trajectories initially coincide before diverging into different regions.The divergence is presented as the point at which a gap emerges during learning.
  • 5 CONCLUSIONS: Across over 100 experiments, nine models, and eight auxiliary-data categories, the study compares VLM effects on manipulation-task VLAs.Experiments span three environments and assess standard VLM competence, auxiliary embodied tasks, and modality choices.
  • 5 CONCLUSIONS: VLM performance on standard VQA benchmarks shows a notable discrepancy with effectiveness when deployed in a VLA.The conclusion identifies this discrepancy as a core gap between current VLM capabilities and embodied-task demands.
  • 5 CONCLUSIONS: Physical-robot experiments are absent because fairness, reproducibility, test efficiency, and cross-hardware comparability are difficult to ensure.The paper instead reports analysis across multiple simulation benchmarks and argues that the visual discrepancy likely reflects task heterogeneity beyond image-level sim-to-real differences.

A.1 MORE DETAILS ABOUT TESING ENVIRONMENTS

The evaluation uses three manipulation benchmarks with distinct simulation settings: novel-scene generalization, real-to-sim transfer, and long-horizon tabletop tasks.

  • A.1 MORE DETAILS ABOUT TESING ENVIRONMENTS: Calvin ABC-D trains on scenes A, B, and C, then tests on scene D with novel object and scene colors.Evaluation uses 1000 five-task sequences after 30k training steps and reports average successfully completed tasks per sequence.
  • A.1 MORE DETAILS ABOUT TESING ENVIRONMENTS: SimplerEnv evaluates policies trained on real-world data in simulation, separating Google Robot and WindowX data-source scenarios.The study selects the more challenging Bridge setting because Google Robot has a smaller visual gap and most policies perform well there.
  • A.1 MORE DETAILS ABOUT TESING ENVIRONMENTS: Libero-Long, also called Libero-10, is the most challenging Libero suite, containing 10 long-horizon tasks with varied objects, scenes, and manipulation types.The benchmark uses a fixed tabletop and spans five task suites overall.

A.2.1 TRAINING SETUPS

Training uses matched model configurations and hyperparameters across methods, with benchmark-specific batch sizes, learning rates, and action-chunk lengths.

  • A.2.1 TRAINING SETUPS: All VLM and action-policy experiments use identical hyperparameters and model configurations within each test environment.Dense VLM experiments use 8 NVIDIA A100 GPUs, while Qwen3VL-30B-A3B uses 32; models receive only a single current-frame image and no state information.
  • A.2.1 TRAINING SETUPS: Calvin uses batch size 128, learning rate 2 × 10^-5, and action chunk size 10 for all models.These settings define the benchmark-specific training setup.
  • A.2.1 TRAINING SETUPS: Simpler and Libero use batch size 512, learning rate 5 × 10^-5, and action chunk size 4.The same settings are applied across models within these benchmarks.

A.2.2 ABOUT CHOICE OF HYPERPARAMETERS

The protocol standardizes hyperparameters while selecting checkpoints and action-chunk sizes through evaluation sweeps, and a learning-rate sweep confirms stable model rankings.

  • A.2.2 ABOUT CHOICE OF HYPERPARAMETERS: Using identical hyperparameters is intended to reflect each model’s peak converged capability rather than differences in tuning treatment.The protocol is designed to decouple performance comparisons from model-specific hyperparameter advantages.
  • A.2.2 ABOUT CHOICE OF HYPERPARAMETERS: The study sweeps action-chunk sizes and checkpoints during evaluation, reporting the best performance among runs.Most best results occur before the final checkpoint, indicating convergence before training ends.
  • A.2.2 ABOUT CHOICE OF HYPERPARAMETERS: A learning-rate sweep over {1e-5, 2e-5, 5e-5, 1e-4} finds highly stable downstream performance with consistent relative model rankings.The sweep covers representative Qwen2.5-VL-3B, Qwen2.5-VL-7B, and PaliGemma-1 models on Calvin.
  • A.2.2 ABOUT CHOICE OF HYPERPARAMETERS: Large identical batch sizes are used because the authors believe they stabilize gradients and prevent models from being disadvantaged by gradient noise.This is an explicit experimental assumption underlying the batch-size policy.

A.2.3 LANGUAGE PROMPTS AND FORMAT

The appendix standardizes prompt formatting across VLMs to support fair comparison, while preserving each model’s native instruction-tuning conventions. For Qwen models, full SFT-aligned prompts marginally outperform simplified prompts, so the former are used thereafter.

  • All VLMs receive the simplest prompt format consistent with their pretraining prompt conventions for fair comparison.
  • Kosmos: Kosmos omits grounding-specific task tokens because VLA action output is not a grounding task.
  • Qwen series: Qwen models use instruction-style formats, motivating comparisons between full SFT-compliant and simplified token concatenation schemes.
  • Qwen series: Scheme 1 includes system and user prompts, whereas Scheme 2 removes conversational prompts and retains essential sequence-construction tokens.
  • Prompt comparison: Prompts matching the SFT format perform marginally better than minimal prompts, so Scheme 1 is applied to QwenVL models thereafter.

A.2.4 INFLUENCE OF IMAGE RESOLUTION

Increasing image resolution alone does not substantially improve VLA performance when the vision encoder is fine-tuned. At higher resolutions, freezing the encoder produces a larger performance gap, while other appendix experiments establish pretrained initialization and evaluation-design context.

  • Resolution settings: 224, 512, and 768 image resolutions are compared on Calvin ABC-D with the vision encoder either frozen or trainable.
  • Resolution findings: Increasing resolution from 224 to 768 does not significantly improve VLA performance when the vision encoder is fine-tuned.
  • Resolution findings: Freezing the vision encoder becomes more damaging at higher resolutions, with the 3B-model gap widening from -1.00 to -1.12.
  • Initialization baseline: Training from scratch substantially degrades performance across both Calvin ABC-D and Simpler-Bridge benchmarks and all tested model sizes.
  • Evaluation context: The evaluation framework compares VLM backbones across three benchmarks using standardized graphical encodings and correlation metrics for fitted relationships.
Loading 2601.03309v2…