Source-linked AI summary
ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver
Wenxuan Song, Ziyang Zhou, Han Zhao, Jiayi Chen, Pengxiang Ding, Haodong Yan, Yuxin Huang, Feilong Tang, Donglin Wang, Haoang Li
TL;DR
Conventional VLAs often disperse visual attention rather than focusing on target regions, limiting precise manipulation. ReconVLA introduces implicit grounding by reconstructing target gaze regions from visual outputs, supported by large-scale robotic pretraining. Experiments in simulation and the real world report superior implicit-grounding performance, precise manipulation, and generalization.
Problem
Conventional VLAs often disperse visual attention instead of focusing precisely on target objects, creating a visual-grounding challenge for precise manipulation.
Method
ReconVLA uses a diffusion transformer conditioned on VLA visual outputs to reconstruct gaze regions corresponding to target manipulated objects.
Results
ReconVLA reports superior performance across simulation and real-world experiments, including precise manipulation and generalization to unseen targets.
Takeaways & Limitations
Implicit grounding directs visual attention toward task-relevant regions while supporting precise manipulation and generalization.
Abstract
from arXiv · showhide
Recent advances in Vision-Language-Action (VLA) models have enabled robotic agents to integrate multimodal understanding with action execution. However, our empirical analysis reveals that current VLAs struggle to allocate visual attention to target regions. Instead, visual attention is always dispersed. To guide the visual attention grounding on the correct target, we propose ReconVLA, a reconstructive VLA model with an implicit grounding paradigm. Conditioned on the model's visual outputs, a diffusion transformer aims to reconstruct the gaze region of the image, which corresponds to the target manipulated objects. This process prompts the VLA model to learn fine-grained representations and accurately allocate visual attention, thus effectively leveraging task-specific visual information and conducting precise manipulation. Moreover, we curate a large-scale pretraining dataset comprising over 100k trajectories and 2 million data samples from open-source robotic datasets, further boosting the model's generalization in visual reconstruction. Extensive experiments in simulation and the real world demonstrate the superiority of our implicit grounding method, showcasing its capabilities of precise manipulation and generalization. Our project page is https://zionchow.github.io/ReconVLA/.
1 Introduction
ReconVLA addresses dispersed visual attention in conventional VLAs through implicit grounding: reconstructing target gaze regions from visual outputs to support fine-grained perception and precise manipulation. The model also uses large-scale robotic pretraining, with experiments reporting improved generalization and manipulation performance.
- Traditional VLA models often disperse visual attention instead of focusing precisely on target objects, risking manipulation of incorrect objects.
- ReconVLA conditions a lightweight diffusion transformer on VLA visual outputs to reconstruct the target manipulated region as implicit visual supervision.The target region is termed the gaze region.
- A curated pretraining dataset contains over 100k trajectories and 2 million data samples from open-source robotic datasets.The dataset supports learning visual generation and reconstruction capabilities.
- ReconVLA uses reconstructive tokens to guide denoising from noisy tokens toward scene tokens representing tokenized gaze-region images.The reconstruction objective complements the action branch, which outputs discrete action tokens.
- Experiments in simulation and the real world report stronger visual grounding, precise manipulation, and generalization, including to unseen targets.The authors also report potential for real-world deployment of VLAs.
2 Related Work
Related VLA work emphasizes action generation, generative planning, or explicit visual grounding. ReconVLA instead supervises visual outputs as an auxiliary task while using implicit reconstruction-based grounding.
- Action-centric Vision-language-action Models: Action-centric VLAs generate executable actions from action supervision, whereas ReconVLA additionally supervises visual outputs to enhance visual perception.
- Generative Methods for Manipulation: Generative manipulation methods use generated images, subgoals, or visual plans to guide robotic control.
- Visual Grounding Methods for Manipulation: Explicit grounding methods provide segmented or cropped target images as additional observations to VLA policies.Examples use external models such as LISA or YOLOv11 to extract target objects.
3 Method
ReconVLA augments a VLA with implicit visual grounding by reconstructing the instructed target region while jointly learning action prediction. Its design combines gaze-region supervision, latent diffusion reconstruction, instruction-conditioned visual tokens, and large-scale robotic pretraining.
- Reconstructive Vision-Language-Action: ReconVLA predicts actions from image tokens, text tokens, and robot proprioception, then maps generated action tokens into executable robot control.The regular VLA formulation uses a vision encoder, tokenizer, language model, and action detokenizer.
- Reconstruction Target: The model reconstructs the gaze region, defined as the target manipulated region, to focus perception on instructed objects and support detailed visual representations.The gaze region can also facilitate sub-task planning by focusing on and switching among different sub-goals in long-horizon tasks.
- Loss Function: ReconVLA jointly optimizes autoregressive action prediction and a visual reconstruction term measured between reconstructive tokens and gaze-region targets.The visual reconstruction process uses low-level gaze-region features rather than cloning explicit RGB values.
- Latent Visual Reconstruction: A visual tokenizer extracts gaze-region latent tokens, while a diffusion-transformer denoiser recovers them from noisy tokens conditioned on language-model visual outputs.The denoiser predicts noise and reconstructs the target latent representation; diffusion timesteps index the denoising process.
- Instruction Conditioning: Instruction tokens are prepended before image tokens so causal attention lets visual tokens fuse information from the task instruction.The interleaved format achieves the intended conditioning without degrading inherent language-modeling capability.
- Visual Pretraining: Pretraining pairs original and cropped target-region images from BridgeData V2, LIBERO, and CALVIN, yielding over 100k trajectories and 2 million samples.Grounding DINO segments the region corresponding to the robot’s instructed interaction target, and reconstruction and action losses are jointly backpropagated before task-specific finetuning.
4 Experiments
Experiments evaluate ReconVLA across simulated and real-world manipulation settings, comparing implicit grounding with alternative paradigms and testing its components, long-horizon performance, and generalization. Results support precise visual grounding, manipulation, and performance on unseen tasks.
- Visual Grounding Comparison: The experiments compare visual grounding paradigms, including explicit grounding, chain-of-thought grounding, and implicit grounding, using the same baseline.The comparison is conducted on CALVIN ABC→D.
- Visual Grounding Comparison: Implicit grounding achieves the highest success rates, while explicit grounding improves spatial comprehension but suffers from visual redundancy and coordinate-based chain-of-thought grounding performs worse.The authors attribute the advantage to directly supervising visual outputs without additional inputs.
- Attention Analysis: ReconVLA concentrates attention on the target object, whereas the baseline attends diffusely or to irrelevant regions, producing inaccurate actions.For placing a watermelon into a yellow bowl, ReconVLA focuses on the watermelon and completes the task.
- Ablation and Precise Manipulation: 79.5% success on “stack block” versus 59.3% for the baseline represents a 20.2% increase from the gazing mechanism.The task requires lifting one block and precisely stacking it on another.
- Ablation and Generalization: Large-scale pretraining substantially improves visual-reconstruction generalization, and reconstructing the gaze region is more effective than reconstructing the entire image.Whole-image reconstruction still exceeds the baseline but is limited in unseen scenarios by pixel redundancy.
- State-of-the-Art Comparison: In long-horizon CALVIN tasks, ReconVLA averages 4.23 of 5 consecutive tasks with 98.0% success on the first task and exceeds OpenVLA by 20.6% and UniVLA by 7.6% on the last ABC→D sub-task.It also surpasses generative methods, including GR-1, by over 20% on the last ABC→D sub-task.
- Real-World Generalization: In real-world experiments, ReconVLA achieves the highest success rate across four tasks and retains successful target grounding when objects are unseen during training.OpenVLA and PD-VLA exhibit nearly 0% success on unseen tasks, while ReconVLA benefits from large-scale mixed-data pretraining.
5 Conclusion
The paper identifies dispersed visual attention as a limitation of traditional VLAs and introduces ReconVLA, which uses implicit grounding to improve attention allocation and manipulation. Large-scale pretraining supports generalization across diverse scenes and unseen objects, with simulation and real-world experiments demonstrating the method’s superiority.
- ReconVLA addresses dispersed visual attention in traditional VLAs through an implicit grounding paradigm.
- ReconVLA accurately allocates visual attention and enhances manipulation skills.
- A large-scale pretraining dataset helps ReconVLA generalize across diverse scenes and unseen objects.
- Extensive simulation and real-world experiments show the superiority of the implicit grounding methods.