PaperlayerView exact revision on arXiv

Source-linked AI summary

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

Yuan Fang, Zechu Li, Haolei Tong, Puze Liu, Georgia Chalvatzaki

arXiv:2609.32069v2cs.RO

TL;DR

Frozen VLA policies typically cannot improve from their own failures, while manual resets and task-success supervision hinder autonomous real-world RL. FIND selects feasible, weakness-relevant tasks from the persistent scene, evaluates outcomes, and improves a deployed policy. Across eight real-world tasks, independent human-assessed success increases from 55% to 71.9%.

  • Problem

    Frozen VLA policies typically cannot learn from their own failures, while manual environment resets and task-success supervision hinder autonomous real-world RL.

    Source: §I · p. 1

  • Method

    FIND selects feasible tasks from the persistent scene according to recent performance, evaluates outcomes from paired images, and uses them for task allocation and policy learning.

    Source: §I · p. 1, §III · p. 3

  • Results

    Independent human-assessed success improves from 55% to 71.9% across eight real-world tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across reversible real-world manipulation tasks, the scene left by one interaction can become the starting point for subsequent practice, reducing reliance on repeated manual resets.

    Source: §V · p. 7

    FIND assumes a predefined task space largely supported by the pretrained VLA and focuses on reversible or mutually compatible task pairs; its task library also limits coverage of deployment scenes.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures. Real-world reinforcement learning (RL) offers a path to continued improvement, yet manual environment resets and task-success supervision hinder autonomous learning. We introduce \textbf{FIND}, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace. FIND reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem: instead of restoring a predefined scene after each rollout, it uses the resulting scene to determine what to practice next. A vision--language agent identifies feasible tasks from a predefined library, prioritizes those with lower recent success rates, and evaluates outcomes using paired pre- and post-execution observations. We instantiate FIND with a frozen $\pi_{0.5}$ VLA and residual off-policy RL. Across eight real-world manipulation tasks, the independent human-assessed success rate improves from $55\%$ to $71.9\%$. A representative run completes 456 autonomous episodes within 6 hours of interaction, requiring 30 scene-recovery interventions and no human-provided reward labels during online learning. Ablations and systematic evaluations further examine key design choices, agent evaluation accuracy, and human intervention requirements. Our website is made publicly available at: FIND.github.io.

I. INTRODUCTION

FIND addresses the difficulty of autonomous real-world VLA learning by selecting feasible practice from the scene left by the previous rollout. It combines performance-aware task selection, outcome evaluation, and policy improvement, and improves human-assessed success across eight real-world tasks.

I. INTRODUCTION

  • VLAs can fail under changes in viewpoint, calibration, contact dynamics, workspace layout, or object geometry, and frozen policies cannot learn from those failures without further human-collected demonstrations and fine-tuning.Real-world RL offers a route to learning through deployment experience, but autonomous practice is hindered by environment resets and task-success supervision.Source: §I · p. 1
  • FIND treats the scene produced by one rollout as the next starting condition, selecting feasible tasks that expose current policy weaknesses rather than restoring a predefined scene.This reframes practice as scene-conditioned task selection.Source: §I · pp. 1–2
  • FIND uses a vision-language agent to identify feasible tasks, prioritizes those with lower recent success, and evaluates outcomes from paired pre- and post-execution observations.The binary outcome informs both subsequent task allocation and policy learning.Source: §I · p. 1, Figure 1 · p. 2
  • FIND closes the loop among scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace.The overview depicts a frozen VLA supplying base actions, a residual policy making corrections through a critic-based gate, and asynchronous actor–critic learning from offline demonstrations and online experience.Source: §I · p. 2, Figure 1 · p. 2
  • Human-assessed success rises from 55% to 71.9% across eight real-world tasks with FIND.A representative six-hour run required 30 scene-recovery interventions and no human-provided reward labels during online learning.Source: §I · p. 2

II. RELATED WORKS

Prior work studies online adaptation of pretrained policies and autonomous or reset-free learning. FIND emphasizes deciding what to practice next through scene-conditioned feasibility, performance-aware allocation, and autonomous evaluation.

A. Real-World Reinforcement Learning for VLAs.

  • Prior VLA adaptation methods primarily improve a pretrained policy once the task and supervision are specified, whereas FIND couples feasibility, performance-aware allocation, and autonomous outcome evaluation during persistent multi-task interaction.The related work describes residual RL and other online adaptation approaches, then distinguishes FIND by its focus on what to practice next.Source: A. Real-World Reinforcement Learning… · p. 2
  • Across both areas, FIND's stated distinction is integrating scene-conditioned task choice and autonomous evaluation with policy improvement during persistent interaction.This distinction complements work that focuses on policy adaptation after task and supervision are specified, or on reset-free practice mechanisms.Source: A. Real-World Reinforcement Learning… · p. 2, B. Autonomous and Reset-Free Robot… · p. 3

B. Autonomous and Reset-Free Robot Learning.

  • Reset-free approaches use learned reset or undo behaviors, compatible task structures, or explicit curricula; FIND frames the autonomous reset-free loop as a VLM-enabled agentic system.The cited approaches include task and reset policies, compatible tasks, curricula, and autonomous practice.Source: B. Autonomous and Reset-Free Robot… · pp. 2–3

C. Agentic Systems for Robot Learning.

Foundation models serve as reasoning and planning components in robot systems, including language-grounded planning, embodied models, and generated programs or rewards. FIND targets agentic real-world RL for self-improving VLA models.

C. Agentic Systems for Robot Learning.

III. METHODS

FIND uses persistent-scene interaction to connect feasible task selection, performance-aware practice, and outcome evaluation. Its implementation pairs this loop with a frozen VLA and residual off-policy RL.

III. METHODS

  • A VLM identifies tasks feasible in the current scene, and a performance-aware curriculum prioritizes them according to recent policy competence.Source: §III · p. 3
  • Paired-image self-evaluation determines task success and uses the outcome to update future task allocation and provide a policy-learning signal.Source: §III · p. 3
  • The implementation uses a frozen VLA with residual off-policy RL trained on offline demonstrations and autonomously collected experience.Source: §III · p. 3

A. Problem Formulation

FIND formulates self-improvement as repeated task selection and learning in a persistent multi-task workspace, where scene changes affect what can be practiced. The formulation connects task execution to a binary outcome and the resulting scene for the next episode.

A. Problem Formulation

  • The workspace admits multiple language-conditioned tasks whose feasibility varies with scene state, and online RL refines pretrained VLA competence rather than discovering behaviors from scratch.Source: A. Problem Formulation · p. 3
  • The task library pairs each manipulation task with a natural-language instruction, and the robot observation combines visual input, proprioceptive state, and the instruction.Source: A. Problem Formulation · p. 3
  • Because the workspace is not restored after each rollout, only tasks feasible in the current scene are eligible, and the physical configuration left by one episode becomes the next episode's workspace state.Source: A. Problem Formulation · p. 3
  • The formulation separates autonomous practice into identifying feasible tasks, choosing what to practice given current competence, and determining whether each attempt succeeds.Source: A. Problem Formulation · p. 3

B. Agentic Self-Improvement System

FIND links persistent-scene task selection, performance-aware allocation, and paired-image self-evaluation in an autonomous interaction loop. The evaluated outcome informs both policy learning and subsequent task allocation.

B. Agentic Self-Improvement System

  • At each iteration, the current workspace determines feasible tasks, and the resulting scene can make a different task eligible without reversing the previous action or restoring a predefined state.Source: B. Agentic Self-Improvement System · pp. 3–4, Figure 2 · p. 4
  • FIND tracks rolling binary outcomes and prioritizes feasible tasks with lower recent success rates; histories start from base-policy evaluations and are updated with online outcomes.Source: B. Agentic Self-Improvement System · p. 4
  • FIND masks infeasible tasks and normalizes the remaining weights, so sampling reflects both scene feasibility and policy competence as the workspace and policy evolve.Source: B. Agentic Self-Improvement System · p. 4
  • A VLM predicts binary task success from the instruction and paired observations captured before and after execution, enabling evaluation of the rollout's state transition.Source: B. Agentic Self-Improvement System · p. 4
  • The same self-evaluated outcome supplies sparse reward for policy improvement and updates task history for future allocation, linking execution, evaluation, data allocation, and learning.Source: B. Agentic Self-Improvement System · pp. 3, 5

C. Policy Improvement via Residual Reinforcement Learning

FIND keeps the pretrained VLA frozen while a residual policy learns bounded action corrections from demonstrations and online experience. Critic gating controls when corrections are executed, and asynchronous collection and optimization support continued training.

C. Policy Improvement via Residual Reinforcement Learning

  • The frozen VLA supplies base actions while a residual policy learns bounded corrective actions from offline demonstrations and autonomously collected experience.The residual actor starts at zero correction, initially reproducing the frozen VLA behavior.Source: C. Policy Improvement via Residual… · p. 5
  • Training mixes offline demonstrations with autonomous trajectories and uses a ten-head REDQ-style ensemble critic, whose mean supports actor optimization and execution gating.Offline base actions are reconstructed by querying the frozen VLA.Source: C. Policy Improvement via Residual… · p. 5
  • Rollout collection and RL optimization run asynchronously, with ordered experience transfer and a pause if the collector gets more than 2000 environment steps ahead.The collector uses the most recently available policy parameters.Source: C. Policy Improvement via Residual… · p. 5
  • The residual learner uses an independent language branch whose projected language-conditioned state is provided with visual features to the actor and critic.A frozen Sentence-BERT encoder and trainable projection produce the language representation.Source: C. Policy Improvement via Residual… · p. 5
  • The critic gate compares predicted values of base and corrected actions, falling back to the pretrained VLA when the critic predicts insufficient benefit.The gating threshold is δ.Source: C. Policy Improvement via Residual… · p. 5

IV. EXPERIMENTS

The experiments assess whether FIND improves a pretrained VLA through autonomous real-world interaction with limited human intervention. They also examine performance-aware task selection and critic gating.

IV. EXPERIMENTS

  • The experiments investigate whether FIND can improve a pretrained VLA through autonomous real-world interaction with limited human intervention.The experimental plan includes the robotic platform, eight manipulation tasks, and demonstration dataset for fine-tuning and replay initialization.Source: §IV · p. 5
  • The evaluation examines how performance-aware task selection and critic gating contribute to improvement.Ablations assess the contributions of the curriculum and critic gate.Source: §IV · p. 5
  • The experiments evaluate autonomous operation and policy improvement, including how task allocation evolves during learning.The setup and dataset are introduced separately from the evaluation and ablation studies.Source: §IV · p. 5

A. Experimental Setup and Dataset

Experiments use a real-world tabletop setup with eight reversible manipulation tasks. The resulting scene from one rollout can provide feasible starting conditions for subsequent tasks.

A. Experimental Setup and Dataset

  • The tabletop setup uses two cubes, a bowl, a drawer, a mug, and a mug tree for eight tasks arranged in four reversible pairs.The pairs are cube insertion/removal, stacking/unstacking, drawer opening/closing, and mug hanging/removal.Source: A. Experimental Setup and Dataset · p. 5
  • The resulting scene from one rollout can provide feasible starting conditions for subsequent tasks.Source: A. Experimental Setup and Dataset · p. 5
  • The system uses two ZED X Mini cameras and fine-tunes π0.5 as the base VLA from real-world demonstrations.Only left RGB streams are used, resized to 224 × 224; the policy also receives language and robot proprioception and outputs an 8-D end-effector pose and gripper command.Source: A. Experimental Setup and Dataset · p. 6
  • Demonstrations stored in LeRobot format are used both to fine-tune the base VLA and initialize the offline replay buffer.Each trajectory includes RGB observations, proprioception, end-effector and gripper actions, language instructions, and terminal information.Source: A. Experimental Setup and Dataset · p. 6

B. Experimental Results

Across eight tasks, FIND improved independent human-assessed performance, while task allocation adapted to recent success and ablations showed complementary roles for its stabilization mechanisms.

a) Autonomous operation

  • In a representative 6-hour run, 426 of 456 episodes proceeded without intervention; 30 required manual scene recovery, a 6.58% intervention rate.The 30 interventions comprised 21 unfavorable object poses, 7 difficult-to-separate configurations, and 2 objects outside reach; the longest uninterrupted run was 48 minutes.Source: B. Experimental Results · p. 6, Table II · p. 6
  • FIND remains dependent on the pretrained policy’s recovery-state coverage when failures leave objects in unusual or tightly coupled configurations.The cited examples include unusual mug poses and tightly coupled cube arrangements where the VLA had limited recovery competence.Source: B. Experimental Results · p. 7

b) VLM agent evaluation

  • GPT-5.6 Terra achieved 96% feasible-task selection accuracy and 100% paired-image success-evaluation accuracy, with the lowest average latency among five evaluated models.The evaluation used 100 scenes and 100 pre/post-execution image pairs.Source: B. Experimental Results · p. 7, Table III · p. 7

c) Overall self-improvement

  • Independent human-assessed success rose from 55% to 71.9% across the five runs’ final checkpoint.The online VLM-evaluated rolling success rate reached (70.4±4.6)% at that checkpoint.Source: B. Experimental Results · p. 7

d) Performance-aware task allocation

  • The curriculum assigns higher probabilities to feasible tasks with lower recent success rates and reduces priority as tasks become consistently successful.The resulting task distribution changes during training as policy weaknesses shift.Source: B. Experimental Results · p. 7

e) Effect of curriculum and critic gating

  • Removing either the performance-aware curriculum or critic gate led to substantially weaker improvement.Without the curriculum, practice was not systematically directed toward current weaknesses; without the gate, residual corrections were applied even when the critic predicted no advantage over the frozen VLA action.Source: B. Experimental Results · p. 7

V. CONCLUSION

FIND uses persistent-workspace interaction and scene-conditioned task selection to support VLA self-improvement. Its current scope depends on pretrained competence, compatible task transitions, and task-library coverage.

V. CONCLUSION

  • FIND enables a pretrained VLA to identify feasible tasks, prioritize current weaknesses, evaluate outcomes, and improve through continued interaction.It treats the resulting workspace state as the next episode’s starting point and uses a performance-aware curriculum and paired-image self-evaluator.Source: §V · p. 7
  • The formulation assumes a predefined task space largely supported by the pretrained VLA, which must have enough prior competence to attempt available tasks.The evaluated task suite also uses reversible or mutually compatible task pairs so one interaction can provide a valid starting configuration for another.Source: §V · p. 7
  • The approach may not directly extend to tasks whose terminal states do not enable useful subsequent interactions.Autonomous practice is also constrained by how well the task library covers workspace configurations arising during deployment.Source: §V · p. 8
  • Future work could expand task spaces, discover practice opportunities, reason over longer skill sequences, and expand the underlying VLA’s capabilities during deployment.These directions could move FIND from identifying weaknesses within an existing repertoire toward expanding what the robot can practice and learn.Source: §V · p. 8

Related papers

Loading 2609.32069v2…