Source-linked AI summary
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Yulin Luo, Hao Chen, Zhuangzhe Wu, Bowen Sui, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Qiuxuan Feng, Jiale Yu, Shuo Gu, Peng Jia, Pheng-Ann Heng, Shanghang Zhang
TL;DR
VLA action prediction requires reliable visual grounding, but existing methods provide limited insight into how visual information is processed within the LLM backbone. DeepVision-VLA analyzes this layer-wise behavior and combines deep visual-feature injection with action-guided pruning, outperforming prior VLA methods by 9.0% in simulation and 7.5% in real-world settings.
Problem
VLA control depends on precise visual interpretation, while many visual-enhancement methods treat the LLM backbone as opaque and provide limited insight into visual grounding during action generation.
Method
DeepVision-VLA uses VL-MoT shared attention to inject multi-level DINOv3 features into deep VLA layers and AGVP to prune irrelevant visual tokens using shallow-layer action-to-vision attention.
Results
9.0% in simulated settings and 7.5% in real-world settings: DeepVision-VLA achieves SOTA performance across ten RLBench tasks and four complex dual-arm manipulation tasks.
Takeaways & Limitations
The results support using shared vision-language attention and task-guided visual selection to strengthen action-relevant visual representations in VLA models.
Takeaways & Limitations
The study uses DINOv3 as its Vision Expert, while evaluating alternative vision experts is left for future work.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose \textbf{DeepVision-VLA}, built on a \textbf{Vision-Language Mixture-of-Transformers (VL-MoT)} framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce \textbf{Action-Guided Visual Pruning (AGVP)}, which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0\% and 7.5\% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
1 Introduction
VLA action prediction depends on integrating visual observations with language, yet visual grounding weakens in deeper backbone layers. DeepVision-VLA addresses this through multi-level vision-feature injection and action-guided pruning, achieving strong simulated and real-world results.
- VLA models map visual observations and language instructions to robot actions, making precise visual interpretation important for robust control and generalization.
- Existing visual enhancements use prompts, auxiliary objectives, additional modalities, and future-state prediction to improve VLA scene understanding and physical modeling.
- Layer-wise analysis finds that attention to task-relevant visual regions is maintained in early layers but weakens in deeper VLA layers during action generation.
- DeepVision-VLA injects multi-level features from a DINOv3 Vision Expert into deep VLA layers through Vision-Language Mixture-of-Transformers shared attention.
- Action-Guided Visual Pruning uses shallow-layer action-to-vision attention to identify task-relevant regions and remove redundant visual tokens before feature projection.
- 9.0% in simulated settings and 7.5% in real-world settings: DeepVision-VLA outperforms prior VLA methods across ten RLBench tasks and four dual-arm manipulation tasks.
2 Related Work
Related VLA research adapts pretrained vision-language models and scales robot demonstration data for control. Visual improvement methods add cues, supervision, or richer modalities to strengthen task understanding and action generation.
- VLA models adapt pretrained vision-language models and scale robot demonstration data to model action sequences from visual observations and language instructions.
- Input augmentation methods add execution trajectories, target-object highlights, or other visual cues to facilitate task comprehension.
- Auxiliary visual supervision encourages attention to important regions, while depth maps, point clouds, and sketches provide complementary spatial information.
3 Methods
The method analyzes how VLA models use visual information across layers, then introduces VL-MoT and action-guided pruning to strengthen deep-layer visual grounding while filtering redundant features.
- 3.1 Preliminaries: VLA models map visual observations and language instructions to robot actions using a visual encoder, LLM backbone, and action decoder.Action representations may predict current actions or short future-action horizons.
- 3.2 Probing the Role of Vision in VLA Models: A layer-wise analysis evaluates visual grounding through attention maps and ROI-token masking across representative VLA architectures and action-generation paradigms.The study uses OpenVLA, π0, and QwenVLA-OFT on 1,500 BridgeV2 trajectories.
- 3.2 Probing the Role of Vision in VLA Models: Shallow layers concentrate visual contributions on task-relevant regions, whereas deeper layers become diffuse and less grounded in relevant visual evidence.This pattern appears across all three analyzed VLA paradigms.
- 3.2 Probing the Role of Vision in VLA Models: Masking ROI visual tokens substantially increases action MSE in early layers, but progressively less in deeper layers, where even complete removal causes only minor changes.The result indicates that task-relevant visual cues are increasingly underutilized along the backbone.
- 3.3.2 VL-MoT Framework: VL-MoT shares attention between a Vision Expert and the deepest VLA layers by integrating their corresponding QKV representations without concatenating full intermediate feature maps.The design preserves separate processing pathways and reduces feature interference during fusion training.
- 3.3.3 Action-Guided Vision Pruning: Action-Guided Visual Pruning averages shallow-layer action-to-vision attention to select task-relevant high-resolution Vision Expert tokens before deep-layer fusion.Top-K selection filters redundant background features, controls computational budget, and injects critical visual-region information.
4 Experiments
DeepVision-VLA is evaluated across simulated and real-world manipulation, with ablations examining its feature integration, visual selection, pruning guidance, and layer references. It achieves strong performance across both settings and under visual perturbations.
- 4.1 Simulation Experiment: DeepVision-VLA achieves an 83% mean success rate on RLBench, outperforming HybridVLA, π0.5, CogACT, and QwenVLA-OFT.It attains the highest success rate on 8 of 10 tasks.
- 4.2 Ablation Study: Using the last 16 DINOv3 layers reaches 88%, outperforming the first 16 layers, uniform DINOv3 sampling, and uniform SigLIP sampling.The corresponding success rates are 61.5%, 85%, and 77%, respectively.
- 4.2 Ablation Study: Action-to-vision attention provides the strongest AGVP guidance, reaching 88% compared with 84% for instruction-to-vision attention and 65.5% for CLS-token guidance.The results associate shallow-layer action tokens with task intent and action-conditioned visual grounding.
- 4.2 Ablation Study: Averaging attention maps across Layers 4-19 reaches 88%, improving over single-layer references whose success rates range from 69% to 87.5%.The multi-layer reference is reported as more robust for suppressing noisy attention to irrelevant regions.
- 4.3 Real-World Experiment: DeepVision-VLA achieves a 91.7% average success rate across four real-world manipulation tasks, surpassing π0.5, Qwen-OFT, and OpenVLA-OFT.It reaches 100% across both stages of pouring coke into a bottle and 95% across both fruit-placement stages.
- 4.3 Real-World Experiment: DeepVision-VLA maintains high success under unseen backgrounds and near-perfect success under altered lighting conditions.Reported examples include 0.90 for Pick fruit Step 1 under unseen backgrounds and 1.00 for Pour coke under unseen lighting.
5 Conclusion
DeepVision-VLA addresses declining visual-token sensitivity in deeper VLA layers through VL-MoT and AGVP, improving visual grounding for robotic action generation.
- DeepVision-VLA observes that visual-token sensitivity progressively diminishes in deeper layers during action generation.
- VL-MoT introduces shared attention between a vision foundation model and the VLA backbone, injecting multi-level visual features into deeper layers.
- AGVP prunes irrelevant visual tokens using shallow-layer attention while preserving critical visual information with minimal computational overhead.
- DeepVision-VLA improves perception-to-action grounding on simulated and real-world robotic manipulation tasks.