Source-linked AI summary
EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Models
Zechen Bai, Chen Gao, Mike Zheng Shou
TL;DR
VLAs trained by static supervised finetuning require many demonstrations and can memorize trajectories without adapting to deployment conditions. EVOLVE-VLA performs test-time training from environmental interaction, using learned dense progress feedback stabilized by accumulative estimation and progressive horizon extension. On LIBERO, it reports gains on long-horizon and 1-shot settings and cross-task transfer to unseen tasks without task-specific demonstrations, while exposing reward-estimator alignment as a limitation.
Problem
Static SFT leaves VLAs dependent on hundreds of demonstrations and brittle trajectory imitation, while deployment lacks the oracle rewards used to train reinforcement-learning policies.
Method
EVOLVE-VLA adapts VLAs through test-time interaction using a learned progress estimator as dense reward, accumulative estimation to smooth noise, and progressive horizon extension.
Results
+8.6% on long-horizon tasks, +22.0% in 1-shot learning, and 0% →20.8% cross-task generalization on unseen tasks are reported on LIBERO.
Takeaways & Limitations
Test-time training enables VLAs to learn from noisy self-generated feedback and develop capabilities such as error recovery and novel strategies beyond demonstrations.
Takeaways & Limitations
Progress-estimator rewards can misalign with rule-based environmental success criteria, causing near-complete failures to receive high rewards and semantic incompleteness to be judged successful.
Abstract
from arXiv · showhide
Achieving truly adaptive embodied intelligence requires agents that learn not just by imitating static demonstrations, but by continuously improving through environmental interaction, which is akin to how humans master skills through practice. Vision-Language-Action (VLA) models have advanced robotic manipulation by leveraging large language models, yet remain fundamentally limited by Supervised Finetuning (SFT): requiring hundreds of demonstrations per task, rigidly memorizing trajectories, and failing to adapt when deployment conditions deviate from training. We introduce EVOLVE-VLA, a test-time training framework enabling VLAs to continuously adapt through environment interaction with minimal or zero task-specific demonstrations. The key technical challenge is replacing oracle reward signals (unavailable at test time) with autonomous feedback. We address this through a learned progress estimator providing dense feedback, and critically, we design our framework to ``tame'' this inherently noisy signal via two mechanisms: (1) an accumulative progress estimation mechanism smoothing noisy point-wise estimates, and (2) a progressive horizon extension strategy enabling gradual policy evolution. EVOLVE-VLA achieves substantial gains: +8.6\% on long-horizon tasks, +22.0\% in 1-shot learning, and enables cross-task generalization -- achieving 20.8\% success on unseen tasks without task-specific demonstrations training (vs. 0\% for pure SFT). Qualitative analysis reveals emergent capabilities absent in demonstrations, including error recovery and novel strategies. This work represents a critical step toward VLAs that truly learn and adapt, moving beyond static imitation toward continuous self-improvements.
1. Introduction
EVOLVE-VLA replaces demonstration-heavy, brittle VLA imitation with test-time adaptation from environmental feedback, using minimal or no task-specific demonstrations. Its learned progress rewards and noise-taming strategies improve performance and support cross-task generalization and emergent error recovery.
- Motivation: Static SFT requires hundreds of demonstrations for new tasks and encourages brittle trajectory memorization that struggles beyond the training distribution.These limitations increase labor costs and impede recovery from execution deviations.
- EVOLVE-VLA: EVOLVE-VLA enables VLAs to continue learning through deployment interaction with only a few demonstrations or none for initialization.The policy explores the target environment, receives feedback, and refines behavior through online reinforcement learning.
- Implications: Test-time adaptation replaces extensive demonstrations with autonomous learning, enabling error recovery, novel strategies, and cross-task generalization through self-directed exploration.Figure 1 illustrates recovery from mistakes developed during test-time training.
- Technical challenge: A learned progress estimator replaces unavailable oracle rewards with dense feedback for test-time reinforcement learning.Progress-based rewards provide continuous feedback, including on failed attempts, and are intended to support sample-efficient learning.
- Technical contributions: Accumulative progress estimation smooths noisy point-wise estimates, while progressive horizon extension lets the policy master simpler subtasks before longer exploration.Together, these mechanisms mitigate estimation noise and make dense but imperfect rewards usable.
- Results: +8.6% on long-horizon tasks, +22.0% in 1-shot learning, and 0% →20.8% cross-task transfer on unseen tasks validate EVOLVE-VLA on LIBERO.The framework also produces emergent capabilities such as error recovery and novel strategies absent from demonstrations.
2. Related Work
Prior VLA research advances multimodal robot control, efficiency, tactile sensing, and reinforcement-learning post-training, but commonly remains dependent on imitation data. EVOLVE-VLA is situated among emerging experience-based approaches that refine policies through interaction rather than only demonstrations.
- Vision-Language-Action Models: VLA research connects large-model semantic priors with multimodal perception and action for generalist robot manipulation.Representative systems include RT, Octo, and OpenVLA.
- Vision-Language-Action Models: Efficiency-focused and tactile VLA methods still rely heavily on manually collected imitation data, limiting labor efficiency and generalization to new tasks and environments.TinyVLA addresses efficiency through parameter sharing and distillation, while other works add tactile modalities.
- RL Fine-Tuning for VLA Models: Recent studies explore reinforcement-learning post-training for VLAs through continual interaction, trajectory-level objectives, and GRPO-like training for embodied navigation.Examples include iRe-VLA, VLA-RL, and OctoNav.
- Concurrent Work: π*0.6: Concurrent work on π*0.6 similarly addresses compounding errors and deployment improvement by learning from autonomous experience rather than only demonstrations.The paper positions EVOLVE-VLA as pioneering academic work in this experience-based direction before π*0.6’s release.
- Framework overview: EVOLVE-VLA’s framework combines environment interaction, rollout generation, progress estimation, GRPO optimization, and progressive horizon extension for test-time training.The overview emphasizes accumulative progress estimation as a source of clean, stable, smooth reward.
3. Method
EVOLVE-VLA enables VLAs to continue learning during deployment through online interaction, using learned task progress instead of oracle rewards. Its method combines diverse rollout generation, GRPO updates, accumulative progress estimation, and mechanisms to stabilize noisy feedback for long-horizon tasks.
- 3.1. Task Definition: EVOLVE-VLA enables test-time adaptation by refining an SFT-pretrained VLA through online interaction in novel scenarios.The policy generates rollouts, receives environmental feedback, and continues learning rather than remaining frozen after imitation training.
- 3.2.1. Online Reinforcement Learning: The framework samples diverse closed-loop trajectories until estimated completion or a maximum horizon, then updates the policy with GRPO using normalized relative rewards.GRPO applies PPO-style clipping without a separate value network.
- 3.2.2. Task Progress Estimation: A learned task-progress reward replaces unavailable oracle success signals and provides dense feedback for differentiating trajectory quality.Progress can be estimated during execution, including failed attempts, supporting sample-efficient learning when successful long-horizon rollouts are initially rare.
- 3.3. Accumulative Progress Estimation: Accumulative progress estimation reduces noisy long-horizon feedback by comparing observations with recent milestone frames and incrementally aggregating critic values.Milestones are sampled at regular intervals, while critic queries occur more frequently; proportional diminishing-returns updates stabilize the completion estimate.
- 3.4. Progressive Horizon Extension: Because noisy rewards make unrestricted long-horizon exploration unstable, the method uses progressive horizon extension to optimize increasingly longer exploration horizons.This strategy allows simpler sub-tasks to be mastered before longer-horizon optimization.
4. Experiments
Experiments show that EVOLVE-VLA improves LIBERO performance, supports low-data and cross-task adaptation, and gains robustness through its progress-estimation and horizon-extension designs.
- Main Results: 95.8% average success rate surpasses π0 (94.2%) and matches UniVLA (95.2%), with gains across all four LIBERO suites including +8.6% on LIBERO-Long.The evaluation applies TTT to OpenVLA-OFT during deployment.
- Low-Data Learning: 61.3% average success rate with one demonstration improves over the 43.6% SFT baseline by +17.7%, with gains across all task suites.The result supports learning from self-generated experience rather than relying solely on extensive expert demonstrations.
- Cross-Task Generalization: 20.8% success on unseen LIBERO-Object tasks improves over 0% for a LIBERO-Long-pretrained policy without task-specific demonstrations.The performance remains below task-specific SFT baselines, reaching 40.1% with one demonstration and 96.6% with 50 demonstrations.
- Ablation Studies: 91.3% success rate and 0.20 F-score with only 32 reward calls show that interval-based accumulative estimation outperforms naive sampling efficiency.Direct two-frame estimates achieve 88.3% success and 0.04 F-score, while uniform sampling requires 96 or 224 reward calls.
- Ablation Studies: 94.4% success on LIBERO-Long with progressive horizon extension exceeds dense-reward training without it by 3.1 percentage points.The method increases rollout length gradually so the policy learns shorter sub-task skills before full trajectories.
- Qualitative Analysis: After TTT, policies recover from failed grasps, adapt to changed object states, and discover strategies absent from demonstrations.These behaviors extend beyond rigid trajectory imitation, although reward-estimator and environment-criterion mismatches remain failure sources.
5. Future Work
Future work targets reward-model generalization, zero-shot transfer, practical real-world training, and safe exploration. The authors also identify efficiency, exploration, and curriculum improvements for more complex tasks.
- Reward-model generalization: Future reward models should better align progress estimates with environment success criteria to reduce mismatch with rule-based rewards.The current accumulative mechanism handles noisy estimates, but does not eliminate semantic mismatch.
- Reward-model generalization: Reward models still benefit from task-specific context, limiting seamless adaptation to entirely novel tasks without task-specific examples.The authors propose training reward models on more diverse manipulation data with better generalization.
- Real-world deployment: Real-world deployment is constrained by online RL training time because physical data collection is slower than simulation.Suggested remedies include sim-to-real reward-model transfer, parallel robot deployment, and more sample-efficient online learning.
- Real-world deployment: Safe real-world exploration requires mechanisms such as action constraints, safety critics, or human oversight because early policies may behave uncontrollably.The authors identify potential damage to robots or environments during early training.
- Future improvements: More sophisticated exploration strategies and curriculum designs could improve sample efficiency for complex, long-horizon manipulation tasks.
6. Conclusion
EVOLVE-VLA enables VLAs to adapt through environment interaction instead of relying solely on static SFT. Using learned progress feedback with accumulative estimation and progressive horizon extension, it improves benchmark performance and produces emergent error recovery.
- Conclusion: +22.0% in 1-shot learning and +8.6% on long-horizon tasks, while unseen-task success rises from 0% to 20.8% without task-specific demonstration training.These results were obtained on the LIBERO benchmark.
- Conclusion: EVOLVE-VLA replaces impractical oracle rewards with a learned progress estimator and uses accumulative estimation plus progressive horizon extension to handle noisy feedback.
- Conclusion: The framework shifts VLAs from rigid trajectory memorization toward continuous adaptive learning through environment interaction.
- Conclusion: Autonomous exploration produces emergent capabilities such as error recovery that are absent from the demonstrations.