PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking VLM Representation for VLA Initialization

Weifeng Lin, Siyuan Huang, Hao Li, Tingwei Chen, Ruichuan An, Xinyu Wei, Jianbo Liu, Hongsheng Li

arXiv:2605.25802v1cs.CV

TL;DR

The paper asks which pretrained VLM representations are useful for initializing VLA policies and studies this through controlled variation of embodied VQA, parameter updates, and robot-data pretraining. It finds that action-relevant signal injection works best when it preserves pretrained representations and matches downstream bottlenecks, with staged LoRA-based robot-data training producing the strongest initialization.

  • Problem

    It is unclear what kind of pretrained VLM representation makes a useful VLA initialization, despite VLMs commonly serving as VLA policy backbones.

  • Method

    The paper controls embodied VQA domains and compositions, Full Finetune versus LoRA updates, and robot-data pretraining within a two-stage VLM-to-VLA pipeline.

  • Results

    The original pretrained VLM representation is a key source of action performance; VQA gains depend on downstream bottlenecks, LoRA is more reliable than Full Finetune, and staged robot-data pretraining is strongest.

  • Takeaways & Limitations

    Effective VLM-to-VLA adaptation should inject action-relevant embodied and robot-trajectory signals while preserving the pretrained representation useful for action learning.

  • Takeaways & Limitations

    The experiments use simulated manipulation benchmarks, so whether the observed bottleneck-alignment and representation-preservation patterns persist on physical robots remains to be validated.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models widely adopt pretrained Vision-Language Models (VLMs) as policy backbones, yet it remains unclear what kind of pretrained VLM representation is useful as a VLA initialization. In this paper, we study VLA initialization as a controlled representation-design problem along three axes: capability-level embodied VQA supervision, parameter-update strategy, and robot-data pretraining. Our experiments show that the original pretrained VLM representation is a key source of action performance. However, embodied VQA adaptation does not yield uniform gains: its benefit depends on downstream bottlenecks, and gains from different capability domains are not simply additive. For update strategy, LoRA provides a more reliable initialization than Full Finetune, indicating that overly reshaping the pretrained representation can weaken VLA initialization. Robot-data pretraining further improves VLA initialization, with the strongest variant obtained by staged LoRA-based training. Together, these findings suggest that effective VLM-to-VLA adaptation should inject action-relevant embodied and robot-trajectory signals while preserving the pretrained VLM representation that remains useful for action learning.

1 Introduction

The paper frames VLA initialization as a controlled design problem: useful adaptation must inject action-relevant signals while preserving pretrained VLM representations. Its experiments show that benefits depend on downstream bottlenecks, update strategy, and robot-data training.

  • VLA initialization asks what kind of pretrained VLM representation provides a useful policy starting point.
  • The study varies embodied VQA domains and compositions, parameter-update strength, and robot-data pretraining.
  • More than 20% performance drops across all benchmarks when policies are trained from scratch, showing the importance of pretrained VLM representations.
  • Embodied VQA gains depend on downstream bottlenecks, and gains from different capability domains are not additive.
  • LoRA is more effective than Full Finetune, while robot-data pretraining improves initialization most strongly through staged LoRA-based training.
  • The resulting principle is to inject action-relevant embodied and robot-trajectory signals while limiting excessive representation drift.

2 Related Work

Related work studies VLM-based VLA architectures, embodied VLM adaptation, and transfer to action policies. These studies motivate embodied adaptation while leaving open which capability signals transfer to manipulation and how they should be composed.

  • VLA systems commonly initialize action policies from pretrained VLMs and use their visual-language representations as priors for action learning.
  • Embodied VLM research targets robotic reasoning, planning, spatial understanding, affordances, referring, and robot-oriented perception.
  • It remains unclear which embodied capability signals transfer to manipulation, motivating controlled comparisons across capability domains and compositions.
  • Prior VLM-to-VLA studies relate VLM choice and embodied-task performance to policy outcomes but leave capability selection, composition, and transfer gaps open.

3 Preliminaries and Study Design

The study separates representation shaping from downstream action learning in a controlled two-stage pipeline. It compares VQA domains, action-head designs, update strategies, robot-data routes, and simulated benchmarks under consistent evaluation settings.

  • The study organizes VLA initialization around embodied VQA domains and compositions, parameter-update strategy, and robot-data pretraining.
  • OpenVLA-OFT uses a lightweight MLP action head, while a π0-style variant uses a diffusion action expert to test initialization patterns across decoder capacities.
  • Seven embodied VQA domains cover spatial relations, grounding, planning and reasoning, camera prediction, and other capability-oriented signals.
  • Stage 1 adapts the base VLM on embodied VQA data, and Stage 2 trains the resulting initialization on action trajectories using a fixed downstream recipe.
  • LoRA updates adapter parameters while preserving most original VLM weights; Full Finetune updates all parameters and permits more aggressive reshaping.
  • Robot-data pretraining compares robot-only, joint robot-data-plus-VQA, and staged VQA-then-robot training routes.
  • Evaluations use consistent observation interfaces, downstream recipes, and protocols across simulated benchmarks with distinct manipulation and control bottlenecks.
  • Libero-10 tests long-horizon single-arm tabletop manipulation, SimplerBridge tests real-to-sim WidowX tasks, and RoboCasa tests diverse bimanual humanoid manipulation.

4 Experiments and Analysis

The experiments treat VLA initialization as controlled representation design across embodied VQA supervision, parameter updates, and robot-data pretraining. Results show that transfer depends on downstream bottlenecks and capability compatibility, while preserving pretrained VLM information generally supports stronger initialization.

  • Experiments and Analysis: Pretrained VLM initialization is critical: training from scratch drops by more than 20% across all benchmarks.The comparison uses the off-the-shelf VLM checkpoint as the Baseline and evaluates both action heads.
  • Single-Domain VQA Adaptation: Embodied VQA transfer depends on both the downstream benchmark and the injected capability, rather than uniformly improving VLA initialization.Libero-10 often improves, SimplerBridge mostly declines, and RoboCasa remains close to the Baseline; Grounding, Egocentric Understanding, and Action-NTP transfer more robustly.
  • Multi-Domain VQA Composition: The {Grounding + Ego} composition performs best, whereas other pairwise and broader compositions saturate or degrade.At a fixed 800k-sample budget, the results indicate that domain gains are not additive and may involve dilution or interference.
  • Preservation vs. Specialization: LoRA consistently provides a stronger VLA initialization than Full Finetune because it better preserves transferable pretrained VLM information.Full Finetune falls below the Baseline in single-domain settings and loses more general VLM capability and downstream VLA performance despite learning the auxiliary task more strongly.
  • Effect of Base VLM Strength: LoRA gains shrink with weaker base VLMs: on Libero-10, the average gain decreases from +3.03% for Qwen3-VL-4B to +1.43% for Qwen3-VL-2B and becomes negative for PaliGemma2-3B.Full Finetune remains below each model’s own baseline across all tested backbones and benchmarks.
  • Synthesis: A practical strategy is to inject action-relevant supervision with constrained adapter updates while starting from a strong VLM and matching signals to the downstream bottleneck.The paper frames initialization as representation design that balances embodied specialization against representation drift.
  • Robot-Data Pretraining for VLA Initialization: Robot-data pretraining improves initialization, with robot-data-only pretraining rising from 49.5% to 52.0% under Full Finetune and reaching 54.0% with LoRA.Adding VQA raises Full Finetune performance to 53.2%, while joint robot-VQA LoRA underperforms robot-data-only LoRA, consistent with competition for adapter capacity.

5 Conclusion

The paper frames VLA initialization as controlled representation design, showing that effective initialization combines action-relevant signal injection with preservation of the pretrained VLM representation.

  • Effective VLA initialization requires both action-relevant signal injection and preservation of the pretrained VLM representation.

A.1 VQA Data Sources

The appendix documents the embodied VQA data and controlled training settings used for Stage-1 adaptation, Stage-2 VLA evaluation, and robot-data pretraining.

  • A.1 VQA Data Sources: Stage-1 embodied VQA data are grouped into seven capability-oriented domains, with 800K samples per single-domain model.Multi-domain compositions use a fixed total budget sampled evenly across selected domains.
  • The experiments hold non-targeted settings fixed within each benchmark and action architecture so differences primarily reflect the initialization factor under study.
  • Stage-1 VLM adaptation uses one epoch of sampled VQA data with AdamW, learning rate 5×10−5, global batch size 128, and cosine scheduling.LoRA and Full Finetune comparisons use the same data budget, optimizer, learning rate, batch size, and epochs.
  • Each Stage-1 checkpoint initializes a VLA policy trained on action trajectories under fixed benchmark-specific Stage-2 settings.
  • Robot-data pretraining compares robot-only, joint robot-data and VQA, and sequential pretraining using AgiBot-World-Beta action supervision.Sequential pretraining starts from the {Grounding + Egocentric Understanding} adapted VLM; evaluation uses Qwen3-VL-4B with OpenVLA-OFT on RoboCasa.

B Analysis of Representation Preservation Strength

The analysis varies LoRA merge strength to test how much embodied VQA adaptation should alter the pretrained VLM before downstream action learning.

  • Analysis of Representation Preservation Strength: The merge-strength diagnostic interpolates between LoRA preservation and stronger representation reshaping by scaling a learned Grounding LoRA update.The study uses Grounding and Libero-10 because this setting shows clear positive transfer.
  • 95.6% is the highest reported Libero-10 success rate, reached at λ = 1.0 after rising from 92.4% at λ = 0 to 94.6% at λ = 0.5.
  • Further amplification reduces Libero-10 success to 95.0% at λ = 1.5 and 92.7% at λ = 2.0.The non-monotonic pattern indicates that stronger injection does not continually improve downstream action performance.
  • Over-amplifying the LoRA update may move the model too far from its pretrained representation and weaken downstream transfer.

C VLM Retention Diagnostics

The retention diagnostic compares auxiliary embodied VQA fitting, general VLM capability retention, and downstream VLA transfer for LoRA and Full Finetune.

  • VLM Retention Diagnostics: The diagnostic evaluates embodied VQA on held-out 1k-example splits and measures general VLM retention alongside downstream VLA transfer.
  • Full Finetune achieves higher embodied VQA scores than LoRA in Grounding and Egocentric Understanding but loses about 18% on average across MMBench and MMStar.
  • Full Finetune falls below the Base VLM in downstream VLA average success rate, whereas LoRA keeps general VLM performance closer to the Base VLM while improving transfer.

D In-Family Validation of Bottleneck-Aligned Transfer

Libero-10-plus confirms that embodied VQA transfer extends beyond a single benchmark split, while domain benefits vary with the downstream action-learning bottleneck. Harder benchmarks do not uniformly eliminate gains; the most useful capability can shift within the same benchmark family.

  • Libero-10-plus increases demands for object localization, task planning, reasoning, and robust scene understanding through added perturbations and noise.
  • All single-domain VQA adaptations improve Libero-10-plus over baseline, with gains ranging from +0.7% to +7.0%.The result supports a broader LIBERO-style transfer pattern rather than a Libero-10-specific effect.
  • Temporal Understanding is strongest on Libero-10, whereas Grounding is strongest on Libero-10-plus, followed closely by Egocentric Understanding.The domain ranking shifts despite both tasks belonging to the same benchmark family.
  • Benchmark difficulty alone does not explain transfer: adaptation is more helpful when the injected capability matches the downstream action-learning bottleneck.The authors contrast this interpretation with a simple easy-versus-hard benchmark explanation.

E Additional Probing Experiments

Frozen-backbone probing tests whether adapted VLM features can support action decoding without further backbone adaptation. Lightweight action heads perform poorly overall, but several capability domains provide comparatively stronger action-related signals.

  • With a lightweight MLP head, frozen VLM representations achieve only 0–2% success across all domains and benchmarks.This indicates that fixed features are nearly unusable for direct action decoding with a low-capacity head.
  • Grounding is strongest on Libero-10 and RoboCasa, while Spatial performs best on SimplerBridge in frozen-backbone probing.
  • Egocentric Understanding and Action-NTP improve over the frozen baseline on most benchmarks with the diffusion expert.These domains also show relatively reliable transfer in the main single-domain initialization results.
  • The gap between frozen probing and main training indicates that effective VLA initialization is not simply direct action decoding from static VLM features.

F Limitations and Future Work

The study’s conclusions are bounded by simulated evaluation, a coarse VQA taxonomy, limited robot-data pretraining, and primarily empirical evidence for representation preservation. Future work targets physical-robot validation, better supervision selection, broader robot-data regimes, and mechanistic diagnostics.

  • Simulated manipulation benchmarks do not fully capture hardware variation, sensing noise, calibration errors, or dynamics mismatch.The authors call for validation of the observed patterns on physical robot platforms.
  • The seven-domain embodied VQA taxonomy is not exhaustive, and prompt formats, filtering, domain definitions, or sampling ratios may change transfer behavior.Future work proposes quality-aware and bottleneck-aware Stage-1 supervision selection.
  • Robot-data pretraining uses one source and focuses on RoboCasa, limiting conclusions about broader robot-pretraining regimes.The balance between action specialization and representation preservation may differ with larger, more diverse data.
  • The representation-preservation interpretation remains primarily empirical despite transfer, merge-strength, behavioral-retention, and frozen-probing evidence.Feature-level diagnostics or causal interventions are proposed to identify preserved, overwritten, or action-converted features.
Loading 2605.25802v1…