PaperlayerView exact revision on arXiv

Source-linked AI summary

ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy

Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, Dongbin Zhao

arXiv:2502.05450v2cs.ROcs.AI

TL;DR

VLA supervised fine-tuning struggles with limited, inconsistent demonstrations, particularly in contact-rich robotic tasks. ConRFT uses unified offline and online consistency-based reinforcement fine-tuning, combining behavior cloning, Q-learning, task-specific rewards, and human interventions. Across eight real-world manipulation tasks, it achieves 96.3% average success after 45–90 minutes of online fine-tuning while improving over supervised methods.

  • Problem

    Limited, inconsistent, and sub-optimal demonstrations restrict state coverage, causing supervised fine-tuning to struggle in real-world contact-rich robotic tasks.

  • Method

    ConRFT uses offline Cal-ConRFT and online HIL-ConRFT under a unified consistency objective, combining behavior cloning, Q-learning, task-specific rewards, and human interventions.

  • Results

    96.3% average success was achieved across eight real-world manipulation tasks after 45–90 minutes of online fine-tuning, with a 144% improvement over supervised methods and 1.9x shorter episode length.

  • Takeaways & Limitations

    The results support a practical use of reinforcement learning for safe and efficient VLA fine-tuning in real-world robotic applications.

  • Takeaways & Limitations

    The task-specific reward classifier is vulnerable to distribution shift during exploration, which can enable reward hacking through inaccurate rewards.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation. However, fine-tuning these models through supervised learning struggles to achieve robust performance due to limited, inconsistent demonstrations, especially in contact-rich environments. In this paper, we propose a reinforced fine-tuning approach for VLA models, named ConRFT, which consists of offline and online fine-tuning with a unified consistency-based training objective, to address these challenges. In the offline stage, our method integrates behavior cloning and Q-learning to effectively extract policy from a small set of demonstrations and stabilize value estimating. In the online stage, the VLA model is further fine-tuned via consistency policy, with human interventions to ensure safe exploration and high sample efficiency. We evaluate our approach on eight diverse real-world manipulation tasks. It achieves an average success rate of 96.3% within 45-90 minutes of online fine-tuning, outperforming prior supervised methods with a 144% improvement in success rate and 1.9x shorter episode length. This work highlights the potential of integrating reinforcement learning to enhance the performance of VLA models for real-world robotic applications. Videos and code are available at our project website https://cccedric.github.io/conrft/.

I. INTRODUCTION

VLA models offer strong representations but supervised fine-tuning remains challenged by limited, inconsistent demonstrations and difficult real-world interaction. ConRFT addresses these constraints through unified offline and online reinforcement fine-tuning with human interventions.

  • Pre-trained VLA policies capture powerful representations but often fall short in complex real-world scenarios, requiring domain-specific fine-tuning.
  • Real-world VLA reinforcement learning requires high sample efficiency and risk-aware exploration because physical interaction in contact-rich environments is costly and safety-constrained.
  • ConRFT combines offline and online fine-tuning under a unified consistency-based objective, integrating supervised learning with Q-learning offline and consistency-policy reinforcement learning online.
  • Cal-ConRFT uses a small set of demonstrations to extract an efficient policy and value function, while HIL-ConRFT uses human interventions for safer, more sample-efficient online adaptation.
  • 96.3% average success was achieved after 45–90 minutes of online fine-tuning across eight real-world manipulation tasks.
  • Reinforcement learning methods for language and vision models often require large interaction datasets, motivating more sample-efficient approaches for robotics.

C. Offline-to-online Methods

ConRFT follows an offline-to-online pipeline: offline data initializes the policy and value function, after which online interaction refines behavior. The approach targets limited demonstration coverage and uses Q-function feedback for task-specific improvement.

  • Offline-to-online reinforcement learning initializes a policy from offline datasets before fine-tuning it through online interactions.
  • Existing offline-to-online methods often assume large, diverse datasets with broad state coverage, a condition rarely met in real-world deployments.
  • Supervised fine-tuning adapts a pre-trained VLA model to downstream tasks using a small set of labeled demonstrations while preserving general feature extraction.
  • SFT struggles in contact-rich tasks because limited, inconsistent, and sub-optimal demonstrations restrict coverage of diverse states.
  • ConRFT couples the VLA policy with a learned Q-function so trial-and-error interactions and task-specific feedback can refine behavior.
  • The ConRFT pipeline performs offline fine-tuning followed by online fine-tuning to optimize robotic policies.

A. Stage I: Offline Fine-tuning with Cal-ConRFT

Cal-ConRFT uses a small offline demonstration set to initialize a VLA policy and Q-function through consistency-based behavior cloning and Q guidance. Its consistency policy is designed to handle inconsistent demonstrations while remaining lightweight for inference.

  • Offline setup: 20–30 demonstrations initialize the pre-trained VLA policy before online reinforcement learning, reducing exploration burden and online training time.The offline stage addresses limited zero-shot generalizability to novel robotic configurations.
  • Offline challenge: Limited state coverage in small demonstration sets can produce poor value estimates and hinder policy generalization to unseen states.This motivates augmenting offline reinforcement learning with additional behavioral supervision.
  • Cal-ConRFT: Cal-ConRFT combines behavior-cloning loss with Q guidance under a unified consistency-based objective for offline VLA fine-tuning.The BC component supplies supervisory signals from demonstrations, while Q learning supports value estimation.
  • Consistency policy: The consistency policy maps Gaussian-noised actions conditioned on the encoded state toward actions from the expert distribution.Its action head is computationally lightweight for efficient inference.
  • Offline outcome: The combined BC and Q losses align value estimates with expert actions, improving offline policy learning and providing a stable initialization for online exploration.The stated motivation is effective learning from few demonstrations, including contact-rich tasks requiring control precision.

B. Stage II: Online Fine-tuning with HIL-ConRFT

HIL-ConRFT continues consistency-policy fine-tuning online while retaining offline demonstrations, adding online replay data and human interventions. The design preserves demonstration alignment during exploration and evaluates methods with matched online episodes across tasks.

  • Online data: HIL-ConRFT retains the offline demonstration buffer and adds a replay buffer for online interaction data during further real-world fine-tuning.The online stage updates the VLA model through the consistency policy while interacting with the environment.
  • Online objective: The online consistency objective uses data from the demonstration and replay buffers and closely mirrors the offline objective for rapid adaptation.The Q-loss term is applied alongside the consistency-based policy objective.
  • Loss balancing: During online training, β typically decreases and η increases, while retaining BC to prevent drastic policy deviations and instability in high-dimensional state-action spaces.The BC term maintains alignment with demonstrations during exploration, especially for contact-rich manipulation.
  • Evaluation: Table I compares SFT and Cal-ConRFT offline baselines with HG-DAgger, PA-RL, and HIL-ConRFT online methods over the same number of online episodes.Metrics include success rates and average episode lengths, with 20 trials reported per task.
  • Human-in-the-loop: Human interventions take control during destructive behaviors, provide corrective actions, and add high-level guidance to the demonstration buffer for safer, more efficient exploration.The interventions are integrated through Human-in-the-Loop learning during online exploration.

V. EXPERIMENT AND RESULTS

The experiments evaluate VLA fine-tuning across eight diverse real-world manipulation tasks spanning placement, contact-rich precision, and dynamic object handling. Learning curves compare success, intervention rates, and episode lengths across representative tasks during online training.

  • Experimental scope: Eight diverse real-world manipulation tasks are used to evaluate fine-tuning effectiveness and efficiency.The tasks include object placement, precise contact-rich manipulation, and dynamic object handling.
  • Task diversity: The task suite includes placing bread, inserting a chair wheel, and hanging a Chinese Knot to represent varied manipulation challenges.These examples cover placement, alignment and insertion, and dynamic object handling.
  • Experimental setup: State observations combine wrist and side RGB images with robot-arm proprioception, including poses, twists, forces, torques, and gripper status.Actions are issued at 10Hz using end-effector delta poses or gripper-inclusive targets.
  • Learning curves: Figure 3 displays running-average success rates, intervention rates, and episode lengths over 20 episodes for four methods across five representative tasks.The caption cautions that interventions can produce successful outcomes, making the actual policy success rate lower than the plotted curve.

B. Experimental Results

Across eight real-world manipulation tasks, ConRFT achieves strong performance with brief online fine-tuning, while existing approaches face difficulties in contact-rich settings and efficient exploration.

  • ConRFT Outperforms Supervised Methods: 96.3% average success rate is achieved after 45–90 minutes of real-world training across all tasks, a 144% improvement over the supervised baseline.ConRFT also outperforms HG-DAgger and PA-RL, which achieve average success rates of 65% and 71.3%, respectively.
  • ConRFT Outperforms Supervised Methods: Human corrections can be sub-optimal and inconsistent, limiting supervised improvement on precise contact-rich tasks such as Insert Wheel and Hang Chinese Knot.In Hang Chinese Knot, variation in correction angles introduces noise and conflicting information that prevents precise dexterous behavior learning.
  • ConRFT Outperforms Supervised Methods: PA-RL improves more stably on simpler tasks but struggles on complex manipulation because limited state coverage prevents its policy-agnostic Q-function from generalizing across positions.The limitation is illustrated by Insert Wheel, where precise alignment and controlled insertion forces are required.
  • ConRFT Outperforms Supervised Methods: ConRFT mitigates inconsistent corrections and limited state coverage by exploring a broader range of states and directly optimizing the policy with task-specific rewards.The paper describes this combination as supporting safe exploration and high sample efficiency in real-world environments.
  • ConRFT Outperforms Supervised Methods: 30.7 average episode length is achieved with HIL-ConRFT, making it 1.9x shorter than offline baselines; HG-DAgger and PA-RL reach 56.3 and 51.1 steps.The paper attributes the shorter episodes to consistency-policy optimization that encourages obtaining rewards more quickly.

2) Fine-tuning VLA Outperforms Training From Scratch:

Training from a pre-trained VLA with offline fine-tuning improves online efficiency and safety compared with reinforcement learning from scratch. Cal-ConRFT also supports faster, more stable online adaptation than SFT.

  • 31.9% average success was achieved by HIL-SERL within the same training duration, whereas training from scratch typically requires extensive interaction and frequent interventions.HIL-SERL can eventually reach optimal policies but usually needs over two hours of online training and exhibits higher intervention rates.
  • Offline fine-tuning from a pre-trained VLA reduces online training time and improves sample efficiency compared with training entirely from scratch.ConRFT accelerates policy convergence and enhances final performance after offline initialization.
  • The comparison in Table III uses 150 demonstrations for DP and SFT, 150 RL-collected demonstrations for RLDG, and 20 demonstrations plus 80–120 rollouts for HIL-ConRFT.All metrics are reported over 20 trials per task.
  • Cal-ConRFT and SFT begin online fine-tuning with similar success rates, but SFT shows higher intervention rates during training.The higher intervention rate indicates severe early-stage policy forgetting for the SFT-trained policy.
  • Q loss in Cal-ConRFT enables quicker and more stable online policy improvement from a small demonstration set.SFT may imitate demonstrations effectively yet require substantial online training to handle states and actions absent from the demonstrations.
  • ConRFT is evaluated with RoboVLM using two VLM backbones while fine-tuning only the action head and freezing visual encoders and the transformer backbone.This evaluates applicability across different VLA model configurations.

VI. LIMITATIONS

The paper identifies reward engineering and frozen representation modules as limitations of the current ConRFT implementation. These choices constrain robustness, generalization, and adaptation to unseen scenarios.

  • A. Sensitivity to Reward Engineering: Task-specific binary reward classifiers can be exploited through reward hacking when exploration states differ from classifier training data.A false-positive classifier response may drive the policy toward incorrect behavior.
  • A. Sensitivity to Reward Engineering: Sparse classifier feedback may slow policy learning even when human interventions are available.
  • A. Sensitivity to Reward Engineering: Reward-driven policies are specialized to task conditions and therefore have limited generalization to new environments.Multi-task dense rewards could improve efficiency but require more sophisticated reward engineering.
  • B. Frozen Encoders and Transformer Backbone: Freezing visual encoders and the transformer backbone prevents online training from refining perception and representations for unseen scenarios.The current implementation fine-tunes only the action head to preserve real-time performance.
  • The conclusion characterizes ConRFT as a practical approach for safe and efficient VLA fine-tuning, while the limitations qualify its current scope.

A. Algorithm Illustration

Algorithm 1 implements ConRFT in two stages: offline Cal-ConRFT initialization followed by online HIL-ConRFT learning and interaction. The online stage combines replayed offline data, new transitions, and human interventions.

  • Stage II: Online fine-tuning with HIL-ConRFT: Stage II begins online policy learning after the replay buffer contains at least 100 transitions.
  • Stage II: Online fine-tuning with HIL-ConRFT: Online updates combine half-sized offline and replay-buffer minibatches into a batch of size B.
  • Interaction Thread: Without intervention, the policy samples an action from πψ and stores the resulting transition in the online replay buffer.
  • Interaction Thread: During intervention, the system executes a human action and stores that transition in the offline dataset D.

B. Task Description, Setup and Policy Training Details

The experiments cover diverse manipulation tasks with 6- or 7-dimensional actions and shared training settings. Tasks include grasping, placement, pressing, insertion, and soft-object manipulation.

  • Setup: The 6-dimensional action space represents end-effector delta pose, while the 7-dimensional space adds gripper control.
  • Setup: Shared training settings use learning rate 3e-4, batch size 256, consistency horizon k ∈[0.002, 80.0], and M = 40 sub-intervals.
  • Task descriptions: Pick Banana requires grasping and placing a banana while preserving it and avoiding basket collisions.
  • Task descriptions: Put Spoon requires controlled grasping and placement of a spoon on a blue towel.
  • Task descriptions: Pick Bread and Put Bread require precise bread alignment and collision avoidance when placing bread into or removing it from a toaster.
  • Task descriptions: Open Toaster requires controlled gripper force to press a button without slipping or applying excessive force.
  • Task descriptions: Insert Wheel is contact-rich, requiring precise pin alignment with tight-tolerance slots and complex contact dynamics.
  • Task descriptions: Hang Chinese Knot involves dexterous manipulation of a soft, dynamic object while maintaining contact, tension, and structure.

C. More experiment results

This section presents hardware setups, camera views, and online learning curves for all real-world manipulation tasks evaluated with HIL-ConRFT.

  • Hardware setup and camera views: Figure 5 illustrates the hardware setup and corresponding camera views for eight real-world tasks.The tasks are Pick Banana, Put Spoon, Open Drawer, Pick Bread, Open Toaster, Put Bread, Insert Wheel, and Hand Chinese Knot.
  • Online training curves: Figure 6 presents HIL-ConRFT policy learning curves across all tasks during online training.The curves cover success rates, intervention rates, and episode lengths.
  • Online training curves: The learning-curve metrics are displayed as running averages over 20 episodes.
Loading 2502.05450v2…