PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflection-Based Task Adaptation for Self-Improving VLA

Baicheng Li, Dong Wu, Zike Yan, Xinchen Liu, Lusong Li, Zecui Zeng, Hongbin Zha

arXiv:2510.12710v3cs.RO

TL;DR

Pre-trained VLAs can struggle to adapt efficiently to specific tasks in new environments, limiting reliable deployment. Reflective Self-Adaptation learns from failures and successes through reflective RL and quality-guided SFT. Experiments report faster convergence and higher final success rates than representative baselines.

  • Problem

    Pre-trained VLAs often have low initial success on new tasks, and in-situ RL adaptation can be sample-inefficient.

    Source: §I · p. 1

  • Method

    Reflective Self-Adaptation combines VLM-generated dense rewards from failure analysis with quality-guided imitation of successful trajectories.

    Source: §I · p. 1, §III · p. 2

  • Results

    Experiments on challenging manipulation tasks show faster convergence and higher final success rates than representative baselines.

    Source: §V · p. 8

  • Takeaways & Limitations

    The framework provides a self-improving approach for autonomous VLA adaptation that learns from failures and successes.

    Source: §V · p. 8

Abstract

from arXiv · show

Pre-trained Vision-Language-Action (VLA) models represent a major leap towards general-purpose robots, yet efficiently adapting them to novel, specific tasks in-situ remains a significant hurdle. While reinforcement learning (RL) is a promising avenue for such adaptation, the process often suffers from low efficiency, hindering rapid task mastery. We introduce Reflective Self-Adaptation, a framework for rapid, autonomous task adaptation without human intervention. Our framework establishes a self-improving loop where the agent learns from its own experience to enhance both strategy and execution. The core of our framework is a dual-pathway architecture that addresses the full adaptation lifecycle. First, a Failure-Driven Reflective RL pathway enables rapid learning by using the VLM's causal reasoning to automatically synthesize a targeted, dense reward function from failure analysis. This provides a focused learning signal that significantly accelerates policy exploration. However, optimizing such proxy rewards introduces a potential risk of "reward hacking," where the agent masters the reward function but fails the actual task. To counteract this, our second pathway, Success-Driven Quality-Guided SFT, grounds the policy in holistic success. It identifies and selectively imitates high-quality successful trajectories, ensuring the agent remains aligned with the ultimate task goal. This pathway is strengthened by a conditional curriculum mechanism to aid initial exploration. We conduct experiments in challenging manipulation tasks. The results demonstrate that our framework achieves faster convergence and higher final success rates compared to representative baselines. Our work presents a robust solution for creating self-improving agents that can efficiently and reliably adapt to new environments.

I. INTRODUCTION

Reflective Self-Adaptation targets the low-efficiency challenge of adapting pre-trained VLAs to new settings, using failures and successes to improve task performance without human experts. Its dual pathways pair failure-derived dense rewards with quality-guided imitation, and experiments report improved performance over representative baselines.

I. INTRODUCTION

  • Pre-trained VLAs can have unacceptably low initial success in new settings, while in-situ RL adaptation is often sample-inefficient.This last-mile adaptation problem is described as a barrier to practical deployment.Source: §I · p. 1
  • Reflective Self-Adaptation asks whether agents can improve in new environments by reflecting on experience without human experts, and proposes a dual-pathway framework.The framework learns from both failures and successes.Source: §I · p. 1
  • Failure-Driven Reflective RL uses VLM causal reasoning to synthesize dense rewards from failure analysis, providing targeted guidance for learning.The method reframes reward engineering as structured, automated reasoning.Source: §I · p. 1
  • Success-Driven Quality-Guided SFT selectively imitates high-quality successes to mitigate reward hacking and cold-start exploration.The pathway uses intrinsic quality assessment and a VLM-driven curriculum.Source: §I · p. 1
  • The framework dramatically improves performance over representative baselines on complex tasks.The introduction reports autonomous task mastery, while the conclusion reports faster convergence and higher final success rates on challenging manipulation tasks.Source: §I · p. 2

II. RELATED WORK

Prior work leaves pre-trained VLAs fixed after training and online fine-tuning faces sparse-reward and stability challenges. This framework uses VLM reasoning in the learning loop to synthesize rewards from failures and prioritize high-quality successful experience.

A. Vision-Language-Action (VLA) Models

  • Pre-trained VLAs generalize zero-shot but remain fixed after pre-training, leaving them unable to adapt in-situ to new environments.Prior VLA approaches include large-scale robotics-data Transformers and VLM-backed models, with discrete-token or continuous action generation.Source: A. Vision-Language-Action (VLA) Models · p. 2

B. Reinforcement Learning with VLA Models

  • Online VLA fine-tuning must learn efficiently from sparse rewards and maintain stability without forgetting strong pre-trained priors.The framework addresses these challenges with failure-derived dense rewards and quality assessment that preferentially selects its best successes.Source: B. Reinforcement Learning with VLA… · p. 2

C. VLMs for High-Level Reasoning in Robotics

  • Where prior VLM uses often plan tasks or judge completed actions, this framework makes the VLM an in-the-loop causal reasoner and reward synthesizer.It analyzes in-situ failures to create dense corrective rewards.Source: C. VLMs for High-Level Reasoning in… · p. 2

III. THE REFLECTIVE SELF-ADAPTATION FRAMEWORK

Reflective Self-Adaptation enables pre-trained VLAs to adapt autonomously to specific manipulation tasks in situ without manual reward engineering or expert demonstrations. Its two pathways learn from failures and successful behavior, and the section introduces their roles and organization.

III. THE REFLECTIVE SELF-ADAPTATION FRAMEWORK

  • The framework pairs failure-driven RL, which analyzes errors to construct targeted dense rewards, with success-driven SFT that imitates high-quality behaviors.The approach is designed to avoid manual reward engineering and expert demonstrations.Source: §III · p. 2
  • The failure loop uses VLM reflection to synthesize dense rewards for PPO updates; the success loop performs prioritized SFT guided by intrinsic quality assessment.Figure 1 depicts these as the framework’s two loops.Source: Figure 1 · p. 3
  • The section moves from problem setting to reflective reward construction with RL, then to quality-filtered SFT and a simplified curriculum when needed.The integrated process is summarized in Algorithm 1.Source: §III · p. 3

A. Problem Formulation and Preliminaries

The section formulates manipulation as an MDP in which a pre-trained VLA policy maps visual observations and language instructions to actions. It identifies sparse environmental rewards as a challenge for exploration and credit assignment.

A. Problem Formulation and Preliminaries

  • Manipulation is formulated as an MDP, with a pre-trained VLA policy taking visual observations and a language instruction as input and outputting actions.The state observation is represented by a high-dimensional image.Source: A. Problem Formulation and Preliminaries · p. 3
  • The policy objective is to maximize expected discounted return over trajectories, with γ as the discount factor.The supplied formulation defines the objective using trajectory rewards across the task horizon.Source: A. Problem Formulation and Preliminaries · p. 3
  • Sparse environmental rewards, such as a binary signal only after task completion, make exploration and credit assignment difficult for RL.The framework is designed to address this by generating dense rewards from the agent’s experience.Source: A. Problem Formulation and Preliminaries · p. 3

B. Failure-Driven Policy Optimization via Reflective RL

The failure-driven pathway uses VLM analysis of failures to construct corrective rewards, then optimizes the VLA policy with PPO. Its modular reward design combines interpretable components to guide learning from identified failures.

Reflective Reward Synthesis

  • The VLM analyzes failures through four stages—causal diagnosis, component selection, relationship identification, and structured configuration—to synthesize a reward function.Failure videos ground the causal analysis, while trajectory data helps set component parameters.Source: 1) The Reflective Reward Synthesis… · p. 3

Modular Reward Architecture

  • A predefined, extensible library supplies parameterized positional, orientational, kinematic, and state-based reward components.The library can also be extended with custom components when existing ones do not fit a correction plan.Source: 2) A Modular Architecture for Robust… · p. 4
  • AND, IF, and OR handlers combine components for concurrent, conditional, and alternative objectives, respectively.The AND handler uses a weighted sum, the IF handler gates a body component, and the OR handler takes the maximum component value.Source: 2) A Modular Architecture for Robust… · p. 4

Reflective Reward Synthesis

  • The VLM’s high-level reasoning is constrained to a robust, verifiable output format that specifies how corrective behaviors are implemented.The design balances semantic reasoning about what to do with structured instructions about how to do it.Source: 2) A Modular Architecture for Robust… · p. 4

Policy Optimization

  • PPO alternates experience collection and policy and value updates, using the synthesized dense reward alongside the sparse environmental reward.The policy update maximizes PPO’s clipped objective, and the value network is updated against advantage targets.Source: 3) Policy Optimization with… · pp. 4–5
  • For the autoregressive VLA, action log-probability sums token-level log-probabilities, enabling policy updates to use reflective feedback to learn behaviors that address identified failures.The PPO clipped objective limits changes from the old policy to stabilize training.Source: 3) Policy Optimization with… · p. 5

C. Success-Driven Policy Refinement via Quality-Guided SFT

The success-driven pathway complements failure-based RL by imitating high-quality successful trajectories and using a conditional curriculum when successes are scarce. Its SFT updates combine experience from the main task and, when active, the curriculum.

Success-Driven SFT

  • Success-driven SFT imitates successful experience to constrain RL exploration and reinforce the task objective, countering reward hacking from proxy rewards.The pathway depends on curating the quality of the success data.Source: C. Success-Driven Policy Refinement… · p. 5

Quality Assessment and Replay

  • Successful trajectories receive quality scores based on reflective reward and trajectory length, and prioritized replay favors efficient, well-executed attempts.The finite replay buffer replaces its lowest-quality trajectory when full.Source: C. Success-Driven Policy Refinement… · p. 5

Conditional Curriculum

  • When main-task success falls below a threshold, a VLM-guided curriculum simplifies environmental impediments while keeping the final goal unchanged.Successes from the simplified task are stored separately to provide a grounding signal when main-task successes are unavailable.Source: C. Success-Driven Policy Refinement… · pp. 5–6

Policy Update

  • SFT uses behavior-cloning loss on trajectories sampled from the main-task buffer and, when active, the curriculum buffer.The combined objective weights PPO and SFT losses to balance exploration and imitation.Source: C. Success-Driven Policy Refinement… · p. 6

IV. EXPERIMENTS

The experiments evaluate in-situ adaptation on the LIBERO benchmark and a custom suite designed to challenge pre-trained policies. They use an OpenVLA-7B starting model prepared for the relevant benchmark tasks through SFT.

Evaluation Benchmarks

  • The evaluation includes all four LIBERO suites and a custom LIBERO-Adapt suite of 10 manipulation scenarios designed to challenge pre-trained policies.The LIBERO suites test spatial relationships, object categories, goal objectives, and long-horizon tasks.Source: §IV · p. 6

Evaluation Protocol

  • Experiments start from OpenVLA-7B, prepared for LIBERO tasks with supervised fine-tuning on each respective suite.The evaluations include quantitative results and qualitative case studies.Source: §IV · p. 6

A. Experimental Setup

The implementation uses GPT-4o for reflective reasoning and collects data in parallel across eight A800 GPUs. It specifies settings for reflection, quality scoring, replay, curriculum activation, and SFT updates.

Implementation Details

  • In the experimental implementation, GPT-4o handles all reflective reasoning tasks, with reflection performed once every five rounds of parallel data collection distributed across 8 A800 GPUs.The trajectory quality score uses weights wreward = 1.0 and wsteps = 0.01; prioritized experience replay uses an exponent of α = 0.6, and DSFT stores up to 50 trajectories.Source: A. Experimental Setup · p. 6
  • The conditional curriculum activates below a 0.1 success rate, and the SFT loss weight is 0.1 with batch size 16.The RL component uses the VLA-RL PPO implementation with consistent hyperparameters.Source: A. Experimental Setup · p. 6
  • The success-driven pathway prioritizes replay using exponent α = 0.6 and limits its SFT replay buffer to 50 trajectories.These settings are reported alongside the trajectory-quality weights of 1.0 for reward and 0.01 for steps.Source: A. Experimental Setup · p. 6

B. Major Experimental Results

Reflective Self-Adaptation achieves higher average success across the four LIBERO suites and learns faster and more stably on LIBERO-Adapt. Qualitative cases show successive failure reflections adding targeted rewards to reach successful execution.

B. Major Experimental Results

  • In two qualitative cases, successive reflections add targeted avoidance or orientation rewards after new failures, leading to successful or collision-free execution.The examples involve transporting a moka pot without tilting and picking the bowl beside a ramekin while avoiding a ketchup bottle.Source: B. Major Experimental Results · pp. 6–7
  • On LIBERO-Adapt, the full framework converges faster and more stably than VLA-RL and Sparse RL, whose performance degrades after an initial peak.The authors attribute stability to success-driven SFT regularizing policy updates and preventing catastrophic forgetting of successful behaviors during RL exploration.Source: Figure 3 · p. 7, B. Major Experimental Results · p. 7
  • Reflective Self-Adaptation achieves a higher average success rate across all four LIBERO suites than representative approaches, including VLA-RL, with π0-FAST as an approximate upper-bound reference.The comparison follows VLA-RL’s evaluation protocol; the method is described as a model-agnostic wrapper for pre-trained VLAs.Source: B. Major Experimental Results · p. 7

C. Ablation Study

Ablations show that both pathways contribute to the integrated framework: reflective-reward-only learning can collapse through reward hacking, while SFT-only learning is stable but suboptimal. SFT guidance also focuses exploration on task-relevant regions.

C. Ablation Study

  • Removing any core component degrades final performance in the LIBERO-Adapt ablation study, supporting the necessity of the integrated design.Table II reports ablation results on the LIBERO-Adapt task suite; the supplied text gives no individual cell values.Source: Table II · p. 7, C. Ablation Study · p. 7
  • With SFT guidance, exploration focuses on task-relevant regions; without it, the agent explores extensively and unfocusedly.Figure 4 visualizes the exploration space projected onto the first two action-space dimensions.Source: Figure 4 · p. 7, C. Ablation Study · p. 7
  • Without Success-Driven SFT, reflective-reward-only learning improves rapidly before catastrophic collapse, illustrating reward hacking.The authors identify the SFT pathway as critical for goal alignment.Source: C. Ablation Study · p. 7
  • Without Failure-Driven RL, Reflective SFT learns stably but plateaus at a suboptimal level, indicating that RL exploration helps discover more effective strategies.Source: C. Ablation Study · p. 7

V. CONCLUSIONS

Reflective Self-Adaptation addresses in-situ adaptation of pre-trained VLAs through learning from failures and successes. Experiments on manipulation tasks show faster convergence and higher final success rates than representative baselines.

V. CONCLUSIONS

  • Reflective Self-Adaptation adapts pre-trained VLA models in situ through a dual-pathway architecture that learns from failures and successes.Source: §V · p. 8
  • The failure-driven pathway uses VLM causal reasoning to synthesize dense rewards from failures, while success-driven SFT imitates high-quality successes to support stability and prevent reward hacking.Source: §V · p. 8
  • On challenging manipulation tasks, the framework achieves substantially faster convergence and higher final success rates than representative baselines.Source: §V · p. 8
Loading 2510.12710v3…