Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization

Nikita Kachaev; Mikhail Kolosov; Daniil Zelezetsky; Alexey K. Kovalev; Aleksandr I. Panov

Source-linked AI summary

Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization

Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov

arXiv:2510.25616v1Oct 29, 2025cs.LGcs.AIcs.RO

TL;DR

VLA action fine-tuning may degrade the visual-language representations and knowledge inherited from pretrained VLMs, raising questions about their retention in embodied settings. The paper diagnoses this degradation with representation and attention analyses plus VL-Think, then introduces visual alignment with a visual teacher; the method improves OOD generalization, while broader recovery is limited under modest data and LoRA constraints.

Problem
It remains unclear whether VLA action fine-tuning preserves the visual-language representations and knowledge inherited from pretrained VLMs.
Method
The paper combines attention-map and representation analyses with the VL-Think diagnostic suite and a lightweight visual alignment method that anchors VLA vision representations to a visual teacher during fine-tuning.
Results
Visual alignment consistently improves out-of-distribution generalization across diverse domains, yielding up to a 10% relative gain over naive SFT.
Takeaways & Limitations
Maintaining visual-language alignment during fine-tuning is important for building VLAs that retain pretrained perceptual knowledge while adapting their action policies.
Takeaways & Limitations
The modest size and diversity of the SFT dataset and limited expressivity of LoRA updates may be insufficient to restore less frequent VL concepts underrepresented in robotics data.

Abstract

from arXiv · show

The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. In this work, we conduct a systematic study of representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, we probe VLA's hidden representations and analyze attention maps, further, we design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. We further evaluate a range of strategies for aligning visual representations and introduce a simple yet effective method that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios. Taken together, our analysis clarifies the trade-off between action fine-tuning and the degradation of VL representations and highlights practical approaches to recover inherited VL capabilities. Code is publicly available: https://blind-vla-paper.github.io

1 INTRODUCTION

The paper studies whether action fine-tuning preserves the visual-language representations inherited from pretrained VLMs. It finds degradation and introduces lightweight visual alignment to retain multimodal understanding and improve OOD generalization.

1 INTRODUCTION: The deployment setting is limited-data supervised fine-tuning of a pretrained VLA for a chosen robotic embodiment and domain, where preserving inherited VL capabilities remains uncertain.The paper frames this as a question of recovering inherited representations during robotic action fine-tuning.
1 INTRODUCTION: Existing retention strategies use auxiliary reasoning, web-scale multimodal co-training, or frozen backbones, but may require heavy supervision, high computation, or architectural constraints.
1 INTRODUCTION: Visual Representation Alignment anchors VLA vision features to strong visual teacher embeddings during SFT, preserving multimodal understanding without added complexity.The method uses precomputed teacher features and a lightweight regularization term.
1 INTRODUCTION: The proposed alignment method improves out-of-distribution generalization by up to 10% relative over naive SFT across Simpler benchmark variations.The comparison is reported across diverse generalization settings in the Simpler-based benchmark.
1 INTRODUCTION: Naive action fine-tuning can degrade VLA visual representations, producing representation collapse, attention sink, and domain-specific forgetting relative to the initial VLM.The study compares hidden representations, attention maps, and VL knowledge transfer across VLM and VLA models.
1 INTRODUCTION: Together, the findings identify a trade-off between action adaptation and VL representation degradation and provide a practical recipe for retaining pretrained perceptual knowledge.

2 RELATED WORKS

Related work establishes VLAs as multimodal robotic models built on pretrained vision-language knowledge and places visual representation alignment within broader representation-learning research. The paper addresses limited evaluation of whether inherited VL capabilities survive action fine-tuning.

2 RELATED WORKS: VLA research extends vision-language pretraining to robotic action prediction, targeting generalization across manipulation tasks and modalities.
2 RELATED WORKS: The Platonic Representation Hypothesis motivates alignment methods by proposing convergence toward shared semantic and perceptual representation spaces across modalities and architectures.
2 RELATED WORKS: VL-Think evaluates transfer of inherited VL capabilities independently of low-level control performance, complementing existing VLA evaluations focused mainly on task execution under distribution shifts.
2 RELATED WORKS: Recent methods align learned representations with strong image, multimodal, 3D, or video backbones to improve quality, grounding, correspondence, or temporal consistency.

3 PRELIMINARIES

The VLA processes image and instruction tokens jointly through a multimodal Transformer, then generates output tokens autoregressively under a causal mask.

The input sequence combines visual tokens with textual instruction tokens before entering the VLM backbone.
Image and text encoders map their inputs into the VLA's shared embedding space of dimension d_e.
The Transformer updates multimodal hidden states through stacked self-attention and feed-forward layers.
At each decoding step, the model conditions on the multimodal input and previously generated output tokens.
A causal mask ensures each final hidden state depends only on the input sequence and earlier output tokens.

4 VL-THINK TASK SUITE

VL-Think is a diagnostic suite that evaluates whether VLA models retain visual-language grounding independently of low-level manipulation complexity. It uses controlled board-selection tasks spanning semantic and symbolic concepts.

The suite measures language-to-visual grounding by requiring correct object placement while holding motion complexity fixed.
Its action-free VLM evaluation counts success only when both the predicted board and target location match ground truth.
Episodes use a reliably grasped carrot and textured boards, with instructions specifying one target concept.
The eight tasks probe shape, color, traffic signs, laundry symbols, weather icons, directions, public information, and numeral parity.

5 VL REPRESENTATIONS ANALYSIS

The analysis probes attention, latent-space organization, and VL task transfer to assess how action fine-tuning affects inherited visual-language representations and knowledge.

5.1 Attention sink: Middle transformer layers 14–24 show the strongest semantically meaningful attention because vision-language fusion is most active there.
5.1 Attention sink: Action fine-tuning produces diffuse or misplaced attention, weakening focus on instruction-referenced objects under OOD conditions.
5.1 Attention sink: OpenVLA's attention maps are noisy and weakly correlated with target objects, unlike Qwen2.5-VL's precise object-aligned attention.
5.2 Representations collapse: OpenVLA embeddings form blurred, overlapping clusters for cup, bottle, and knife, unlike the separated clusters of the compared VLMs.
OpenVLA–7B shows substantial VL success-rate drops relative to PrismaticVLM across nearly all domains, especially symbolic and abstract categories.

6 METHOD

The method aligns intermediate VLA visual representations with features from a frozen pretrained vision teacher while retaining the standard action-learning objective. The alignment objective is presented as a lightweight regularizer for OOD robustness.

6.1 Visual representation alignment: Visual Representation Alignment regularizes VLA embeddings toward a frozen pretrained vision teacher's generalized visual representations.
6.1 Visual representation alignment: A projector maps VLA visual tokens into the teacher feature dimension before patch-wise similarity is computed.
6.2 Objective: The total loss combines the autoregressive VLA action objective with a weighted alignment term.
The proposed alignment objective yields consistent OOD gains over standard SFT and frozen-encoder baselines across Semantic, Vision, and Execution axes.
6.2 Objective: The frozen teacher remains a fixed perceptual reference while gradients update the VLA visual and multimodal components.

7 EXPERIMENTS

The experiments evaluate VLA generalization across vision, semantics, and execution shifts, while probing whether action fine-tuning preserves visual representations. Visual alignment improves OOD performance and semantic feature quality, whereas naive SFT degrades representations and domain transfer.

Evaluation Setup: The evaluation uses held-out objects, receptacles, textures, distractor backgrounds, randomized seeds, and linear probing to test OOD robustness and representation quality.Models are compared under matched training budgets and hyperparameters.
OOD Evaluation: Visual alignment consistently improves generalization across Semantic, Vision, and Execution axes, while freezing the visual encoder yields near-zero performance.The comparison covers robustness to instruction variation, visual shifts, texture changes, and background perturbations.
VL-Think: Action fine-tuning causes marked VL degradation across nearly all VL-Think domains, with the largest declines in symbolic and abstract categories; Color remains comparatively preserved.The benchmark contrasts OpenVLA–7B after action fine-tuning with its pretrained VLM counterpart.
Linear Probing: OpenVLA Align achieves stronger ImageNet-100 probing features than both pretrained and naive-SFT variants, surpassing the pretrained baseline.Naive SFT substantially reduces probing accuracy, whereas alignment strengthens semantic consistency and transferability.
VL-Think: Alignment partially mitigates domain forgetting, improving Color and Shape enough to surpass the PrismaticVLM upper bound while leaving other domains mostly unchanged.The result indicates that gains are uneven under constrained fine-tuning settings.

8 ABLATIONS

The ablations identify which teacher, alignment location, projector, and loss configuration most effectively preserve visual representations. Stronger teachers and deeper alignment choices provide the most reliable gains, with a frozen MLP projector and low-weight cosine loss performing best.

Teacher Models: C-RADIOv3 achieves the best overall results among evaluated teacher vision models, supporting stronger teachers as more stable alignment anchors.The comparison includes DINOv2, SigLIP, C-RADIOv3, and Theia.
Alignment Paradigms: Backbone2Enc consistently outperforms Enc2Enc, indicating that representation degradation is concentrated in middle-to-late fusion layers rather than early encoder layers.Regularizing deeper representations preserves visual-semantic consistency while allowing lower layers to adapt.
Projection Methods: A frozen MLP projector provides the most reliable and robust alignment across evaluation dimensions.The authors hypothesize that freezing prevents the projector from absorbing the alignment objective without meaningful representation changes.
Layer Selection: Middle-layer alignment is motivated by the role of those layers in VL fusion and semantic grounding, whereas early and late layers emphasize low-level features and action prediction.The ablation compares alignment across different layer groups.
Loss Functions: Cossim loss with λ=0.2 yields the most stable and consistent improvements across tested loss functions and weights.This setting constrains representation drift without overpowering the task objective.

9 CONCLUSION

The paper shows that robotic action fine-tuning can degrade VLA visual-language understanding and representation quality. A lightweight visual alignment method anchors the VLA to a pretrained visual teacher and improves OOD generalization, although the study focuses on fine-tuning rather than full-scale pretraining.

Conclusion: The study finds that action fine-tuning degrades VL understanding and representation quality in VLA models.The authors use VL-Think diagnostics, attention analyses, and linear probing to expose this degradation.
Conclusion: The proposed lightweight Visual Alignment method improves OOD generalization across novel objects, unseen scene compositions, texture and lighting changes, and instruction paraphrases.The method anchors the VLA to its pretrained visual teacher.
Conclusion: The study focuses on fine-tuning rather than full-scale pretraining because of compute constraints.The authors position scalable robotic pretraining as future work.

A APPENDIX

The appendix supplies technical details, extended results, and supplementary ablations for the proposed alignment method. Its purpose is to improve transparency, reproducibility, and interpretability of the main findings.

Appendix: The appendix contains additional technical details, extended results, and supplementary materials supporting the main findings.It covers alignment strategies, projector architectures, teacher models, layer selection, and alignment coefficients.
Appendix: The supplementary ablations are intended to enhance transparency, reproducibility, and interpretability of the visual alignment method.The appendix complements the paper’s core experiments rather than introducing a separate method.

A.1 Training Hyperparameters

The appendix specifies controlled training, alignment, probing, and ablation configurations for evaluating visual representation quality and design choices. It combines qualitative attention analysis with frozen-feature linear probing and matched statistical comparisons.

Training configuration: Training configurations keep non-alignment hyperparameters identical across methods and variants, isolating alignment-specific settings.Table 9 gives the best OpenVLA visual-alignment fine-tuning configuration.
Attention analysis: Attention maps compare Qwen2.5-VL, OpenVLA SFT, and OpenVLA Align across middle transformer layers where vision-language fusion is strongest.The comparison targets visual grounding for queried objects such as hamburgers and baguettes.
Attention analysis: OpenVLA SFT shows diffuse, spatially inconsistent attention, whereas OpenVLA Align restores sharp, localized, object-centered focus on task-relevant regions.The authors attribute this recovery to mitigation of the attention sink introduced by naive fine-tuning.
Representation probing: Linear probing uses frozen patch embeddings, mean-pooled into a single vector, with one linear classifier and fixed hyperparameters across models.Evaluation reports top-1 accuracy on a reduced ImageNet-100 setup using the same seed and data split.
Alignment ablations: Ablations vary one design component at a time, while paired Wilcoxon tests compare matched per-seed success rates across Semantic, Vision, and Execution environments.The statistical comparisons use 128 shared random seeds and a one-sided alternative B > A.
Alignment ablations: Projector ablations compare cosine, orthogonal, random Fourier feature, whitening-affine, and spectral-norm mappings for visual alignment.The appendix describes the defining transformation used by each projector family.