Source-linked AI summary
EXIMO: VLM Guided Exploration of VLA Policies
Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin Riedmiller
TL;DR
Robot-policy adaptation must address the cost of teleoperation and the sample inefficiency of RL for long-horizon tasks. EXIMO combines VLM-guided exploration, imitation from successful orchestrated trajectories, and residual online RL, and it significantly outperforms the compared baselines in sample-efficiency and performance.
Problem
EXIMO addresses adapting pretrained VLAs to new long-horizon tasks without additional teleoperation, where RL is notoriously sample-inefficient.
Method
EXIMO uses VLM orchestration to collect successful task data, supervised fine-tuning to distill it into the VLA, and residual online RL for further fine-tuning.
Results
EXIMO significantly outperforms the compared baselines in sample-efficiency and performance across manipulation tasks.
Takeaways & Limitations
VLM-guided exploration can be combined with VLA sensory-motor skills to improve adaptation to new robotic tasks without additional teleoperation.
Takeaways & Limitations
The current setup assumes access to ground-truth success detectors; VLM-based success detection is left for future work.
Abstract
from arXiv · showhide
How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.
Introduction
EXIMO targets sample-efficient adaptation of pretrained VLAs to new, long-horizon tasks without additional teleoperation. It combines VLM-guided exploration, supervised imitation, and online residual reinforcement learning, outperforming the compared baselines on 22 simulated manipulation tasks.
- RL is especially sample-inefficient for long-horizon robotic problems, while collecting large teleoperation datasets requires substantial human effort.
- The target setting is a pretrained VLA that performs atomic skills but struggles with new compositional tasks requiring skill chaining or reasoning beyond its training distribution.
- EXIMO orchestrates the VLA with a VLM, distills successful orchestrated episodes through supervised fine-tuning, and then applies online residual reinforcement learning.
- 22 simulated manipulation tasks evaluate EXIMO, covering reasoning about objects and chaining skills to achieve success.
- EXIMO outperforms the SOTA base VLA, the base VLA with additional RL fine-tuning, and the base VLA combined with VLM orchestration.
Related Work
Prior work uses foundation models for planning, exploration, and RL, but EXIMO focuses on sample-efficient fine-tuning of pretrained, natural-language-instructable VLAs. Its VLM orchestrates the VLA directly, avoiding additional grounding during exploration.
- Foundation Models as High-level Planners: LLM-based robotic planners decompose long-horizon tasks, but grounding generated actions into permissible robot actions remains a central challenge.
- Foundation Models as High-level Planners: Recent VLA policies can interact with robots through natural language, reducing the grounding challenges associated with LLM planners.
- Foundation Models as High-level Planners: EXIMO studies sample-efficient fine-tuning of natural-language-instructable SOTA VLAs on challenging robotic tasks.
- Foundation Models for Exploration: Earlier exploration work used LLM plans to train a vision policy from scratch via RL, whereas EXIMO fine-tunes pretrained VLAs instructed directly in natural language.
- Foundation Models for Exploration: VLMs and LLMs have also guided exploration by measuring novelty or interestingness, proposing goals, learning intrinsic rewards, or selecting skill sequences.
- RL Finetuning of VLAs: Large diffusion and transformer VLAs that predict action chunks make direct RL training particularly challenging, motivating alternative RL fine-tuning strategies.
Method
EXIMO combines VLM-guided closed-loop exploration with imitation learning and residual online RL to adapt a pretrained VLA to new tasks. Successful orchestrated trajectories are distilled into the VLA, while residual control supports subsequent RL fine-tuning.
- EXIMO is a three-step training pipeline: Explore, Imitate, and Optimize.
- Explore: VLM-Guided Data Collection: The initial GROD VLA is a 3B vision-language-action model that receives a state and natural-language goal and returns an action.
- Explore: VLM-Guided Data Collection: The VLM decomposes a natural-language goal into intermediary goals that the VLA can execute, adapting those goals from the history of observed states in closed loop.
- Explore: VLM-Guided Data Collection: Successful VLM-orchestrated rollouts are selected using the environment’s ground-truth success detector and stored for supervised fine-tuning.
- Imitate: Supervised Finetuning on VLM Orchestrated Data: During imitation, the VLA learns actions conditioned on the actual task goal rather than the VLM’s intermediary goal, distilling orchestration into one policy.
- Optimize: Finetuning via Online RL: The online RL stage trains a residual policy whose action is added to the VLA action, avoiding direct RL optimization of the large diffusion-based VLA.
- Optimize: Finetuning via Online RL: Sparse success rewards make residual RL difficult, but non-trivial VLA success rates improve residual-policy learning and exploration.
Experiment
EXIMO is evaluated by separately testing VLM-guided exploration, supervised distillation, and online RL across manipulation tasks. VLM orchestration improves exploration efficiency, SFT on orchestrated data further improves the VLA, and RL adds additional gains.
- Setup: EXIMO ablates VLM orchestration, supervised fine-tuning, and online RL across 22 manipulation tasks in simulation.The evaluation uses success rate and time-to-success, with experiments designed to test each stage of the method.
- Explore: VLM orchestration significantly increases GROD’s success rate while maintaining similar time to success.The gain is especially visible on long-horizon skill-chaining tasks and reasoning tasks; orchestrated episodes are also shorter.
- Explore: VLM orchestration produces higher-quality episodes with higher success rates and shorter episode lengths, making data collection more efficient.The collected episodes are subsequently used to investigate efficient fine-tuning of the base VLA.
- Imitate: Filtered SFT on successful VLM-orchestrated data drastically improves the base VLA and outperforms both orchestrated and non-orchestrated agents.Distillation compiles multi-step VLM-guided behavior into a single policy conditioned directly on the task goal.
- Optimize: GROD + SFT starts and converges at higher success rates than the base GROD model, despite the base model receiving significantly more online-RL environment steps.The same behavior is observed for time-to-success, supporting the value of higher-quality orchestrated data.
- Optimize: Online RL further boosts success rates consistently across tasks, while SFT-only performance exceeds the RL-finetuned base VLA on several tasks.These results support both the optimize phase and the value of VLM-orchestrated data distillation.
Conclusion
EXIMO combines VLM semantic guidance with VLA sensorimotor skills to adapt policies to new manipulation tasks. Across the ALOHA benchmark suite, it improves sample-efficiency and performance over the evaluated baselines.
- Conclusion: EXIMO combines VLM general knowledge with VLA sensory-motor skills by breaking complex tasks into executable intermediary steps.The VLA is then improved by distilling the resulting trajectories and further fine-tuning with off-policy residual RL.
- Conclusion: Across several ALOHA tasks, EXIMO significantly outperforms the baselines in both sample-efficiency and performance.The conclusion summarizes the combined effect of VLM-guided exploration, trajectory distillation, and residual RL.
Future Work
The paper identifies extensions for making EXIMO more autonomous and broadening its distillation setting. These include replacing ground-truth success detection, automating resets, and extending beyond off-policy distillation.
- Future Work: The current setup assumes access to ground-truth success detectors, while future work would use VLMs for success detection and reward modeling.The authors also propose using VLMs to orchestrate environment resets by having agents undo reversible tasks.
- Future Work: A fully autonomous learning loop is proposed in which VLMs provide reward modeling, orchestration, and resets for reversible tasks.This remains a future direction rather than part of the current setup.
- Future Work: The current framework is restricted to off-policy distillation, reusing exploration trajectories for supervised fine-tuning.Extending EXIMO to on-policy distillation methods is identified as a promising direction.
- Future Work: After SFT on VLM-orchestrated data, SFT + RL outperforms both SFT-only and RL-finetuned base GROD across tasks.This comparison is reported in Figure 5 and motivates the continued use of online RL after imitation.
Additional Experiments
The experiments compare free-form and grounded VLM orchestration, then test whether orchestrated data benefits residual-policy offline and online reinforcement learning. Free-form and Pick&Place orchestration perform comparably, while residual-policy methods struggle to transfer benefits from orchestrated data to evaluation without the VLM.
- VLM Orchestration: Free-form natural-language instructions and restricted Pick&Place commands lead to comparable performance across five tasks.GROD can follow free-form VLM instructions without additional grounding.
- Residual Policy Distillation: The residual policy improves with orchestrated data during offline RL but learns significantly more slowly than pure online RL after transitioning online.The authors attribute this to distribution shift between offline VLM distillation data and online RL rollouts.
- Online RL with VLM Orchestration: Using the VLM during only a fraction of rollout episodes is intended to reduce distributional shift between training and evaluation without the VLM.This procedure combines VLM-assisted exploration with on-policy data from the residual policy.
- Online RL with VLM Orchestration: The no-VLM method performs best across tasks during evaluation, although VLM orchestration produces significantly more success during initial data collection.The residual policy does not benefit from tuples that correct VLM-orchestrated actions rather than the base GROD action.
Implementation Details
The evaluation uses 22 manipulation tasks spanning compositional, reasoning, spatial, and chained-skill requirements. The implementation prompts a VLM with the task, observations, and instruction constraints to produce natural-language instructions for the VLA.
- Task Suite: The evaluation suite contains 22 manipulation tasks with natural-language goals labeled T2–T23, while T1 denotes the average across tasks.The tasks include dish placement, semantic reasoning, spatial caddy placement, and chained caddy tasks.
- Task Suite: Chained tasks require placing multiple tools into a caddy sequentially, and dish-placement tasks chain multiple pick-and-place skills.The suite therefore tests both compositional skill chaining and spatial or semantic interpretation.
- Task Suite: Reasoning tasks require inferring an unnamed target object from a semantic description, while left/right caddy tasks require spatial understanding.These requirements lie outside the base VLA’s training distribution.
- VLM Prompting: At each step, the VLM prompt is filled with the overall task goal, an observation description, and constraints on the instructions it may emit.The observation and instruction-constraint sub-templates are shown in Figure 10.