Source-linked AI summary
A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
Shaopeng Zhai, Qi Zhang, Tianyi Zhang, Fuxian Huang, Haoran Zhang, Ming Zhou, Shengzhe Zhang, Litao Liu, Sixu Lin, Jiangmiao Pang
TL;DR
Real-world VLA reinforcement learning is limited by sparse, task-specific rewards and inefficient exploration. The paper introduces VLAC, a unified model that estimates pairwise progress and generates actions within an asynchronous real-world RL framework, with optional graded human intervention. Across four manipulation tasks, success rises from approximately 30% to 90% within 200 episodes, while human intervention improves exploration efficiency by approximately 50% and reaches up to 100% success rate.
Problem
Real-world VLA reinforcement learning lacks reliable, transferable intermediate rewards, while collecting expert trajectories is costly and time-consuming.
Method
VLAC unifies critic and policy functions by mapping paired observations and task descriptions to signed progress deltas while also generating actions.
Results
Across four diverse real-world manipulation tasks, success improves from approximately 30% to 90% within 200 episodes; human intervention improves exploration efficiency by approximately 50% and reaches up to 100% success rate.
Takeaways & Limitations
VLAC supports real-world self-improvement from both successful and failed trajectories, with graded human intervention providing an additional exploration aid.
Takeaways & Limitations
Human intervention remains heuristic and operator-specific, with no standardized quantitative triggers for competence plateaus or failure-mode coverage.
Abstract
from arXiv · showhide
Robotic real-world reinforcement learning (RL) with vision-language-action (VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient exploration. We introduce VLAC, a general process reward model built upon InternVL and trained on large scale heterogeneous datasets. Given pairwise observations and a language goal, it outputs dense progress delta and done signal, eliminating task-specific reward engineering, and supports one-shot in-context transfer to unseen tasks and environments. VLAC is trained on vision-language datasets to strengthen perception, dialogic and reasoning capabilities, together with robot and human trajectories data that ground action generation and progress estimation, and additionally strengthened to reject irrelevant prompts as well as detect regression or stagnation by constructing large numbers of negative and semantically mismatched samples. With prompt control, a single VLAC model alternately generating reward and action tokens, unifying critic and policy. Deployed inside an asynchronous real-world RL loop, we layer a graded human-in-the-loop protocol (offline demonstration replay, return and explore, human guided explore) that accelerates exploration and stabilizes early learning. Across four distinct real-world manipulation tasks, VLAC lifts success rates from about 30\% to about 90\% within 200 real-world interaction episodes; incorporating human-in-the-loop interventions yields a further 50% improvement in sample efficiency and achieves up to 100% final success.
1. Introduction
VLAC addresses weakly transferable, task-specific reward design in real-world VLA reinforcement learning by unifying dense progress criticism with action generation. Across four manipulation tasks, the framework improves autonomous success and further benefits from graded human intervention.
- Motivation: Real-world RL uses successful and failed trajectories for autonomous exploration, but sparse or task-specific rewards leave intermediate progress unreliable.These limitations impede sample efficiency and motivate denser, more generalizable progress feedback.
- VLAC approach: VLAC takes paired image observations and a task description, then outputs a signed progress delta as a reward while generating actions in one autoregressive architecture.It is built on InternVL and supports zero-shot and in-context task progress prediction and action generation.
- VLAC approach: More than 4,000 hours of language-annotated manipulation data, additional negative samples, and auxiliary perception tasks train VLAC for progress estimation and action generation.The data combines temporal progress labels with detection, segmentation, grounding, and coarse 3D or contact cues.
- Results: VLAC separates positive and negative progress sufficiently to provide a reliable reward signal, while stronger critic training improves downstream action generation.The critic and actor roles are therefore evaluated together rather than as isolated components.
- Results: From approximately 30% to 90% success within 200 episodes, the actor improves autonomously across four diverse real-world manipulation tasks.A graded human-in-the-loop mechanism further improves exploration efficiency by approximately 50% and reaches up to 100% success rate.
2. Related Work
Related work spans real-world RL, dense general reward modeling, and RL post-training for heterogeneous VLA architectures. The paper positions VLAC against task-specific rewards, limited use of failures, fragmented integration strategies, and open requirements for joint progress, failure, perception, and action understanding.
- Real-world RL: Many online real-world RL methods reinforce successful behaviors or use hindsight relabeling, leaving failed attempts underexploited and limiting data efficiency.The discussion focuses on manipulation and contrasts these approaches with learning from both positive and negative trajectories.
- RL post-training for VLAs: VLA RL integration remains fragmented because architectures vary in perception-language fusion, action representation, formatting, and progress or value interfaces.For post-training, discrete autoregressive actions commonly use token-level PPO, whereas diffusion and flow-matching policies require filtering, weighting, or specialized gradient strategies.
- General reward models: Real-world tasks often provide only terminal success/failure or sparse segmented rewards, creating exploration-depth and credit-assignment challenges.Dense, temporally consistent progress signals are presented as a direct route toward improved sample efficiency.
- General reward models: Existing reward approaches include shared state-goal embeddings, synthesized goal images, and learned progress embeddings for defining task-related reward.These approaches differ in whether reward comes from similarity, goal-image differences, or temporal progression constraints.
- General reward models: Open questions include fine-grained low-noise progress estimation, failure and deviation discriminability, and tight integration of multimodal perception with action generation.These requirements define the paper’s target beyond generic task-agnostic scoring.
- RL post-training for VLAs: Strong pretrained capability and offline-to-online strategies use human or demonstration data to reduce early exploration cost while addressing value underestimation and distributional drift.These methods commonly combine value optimization with behavioral-cloning regularization or demonstration-buffer initialization.
- VLAC positioning: VLAC training combines public robotic, human, self-collected, and image-understanding datasets, with pairwise video samples supporting progress understanding and action generation.The overview emphasizes generalization to new entities, scenarios, and tasks, including zero-shot manipulation and dense feedback for data refinement.
3. VLAC model and Real-World RL Framework
The paper presents VLAC as a unified critic-policy model trained through pairwise progress understanding and designed for cross-scenario, cross-task generalization. Its real-world framework combines action generation, progress estimation, and self-improvement from heterogeneous trajectories.
- VLAC model: VLAC jointly generates actions and delta task progress from pairwise intermediate states, supporting in-context learning and cross-scenario, cross-task generalization.The pairwise module accepts arbitrary intermediate-state images and is agnostic to data-collection strategy and segment starting points.
3.1. Vision-Language-Action-Critic Model
VLAC models task progress from paired observations and language while jointly supporting action generation, using heterogeneous human and robot data for cross-task generalization. Its critic outputs progress and completion signals, and its action formulation is designed for semantic and scene generalization and exploration.
- VLAC critic learning: VLAC uses paired images and a task description to estimate relative progress, with positive values indicating advancement and negative values indicating regression.The formulation is local to sampled frame pairs and is independent of trajectory start points or collection strategy.
- VLAC critic learning: Training combines more than 3000 hours of human data with 1200 hours of public robotic manipulation data and over 15 hours of self-collected manipulation data.This combination provides action control, progress assessment, and task completion verification while alleviating robot-data scarcity.
- VLAC critic learning: Four sampling strategies improve progress robustness by filtering static pairs, balancing forward and reverse samples, balancing completion states, and mismatching task descriptions.Cross-sampling uses a 5% probability for descriptions unrelated to the current trajectory and assigns zero progress.
- VLAC critic learning: In-context learning enables progress estimation from a single reference example, optionally using the current trajectory start to align with absolute progress.References may come from robot or human demonstrations and provide scene and task guidance.
- VLAC action learning: Actions are generated autoregressively as numeric delta end-effector poses, leveraging semantic representations while remaining independent of embodied entities.The formulation supports semantic and scene generalization, and action diversity within a reasonable range benefits reinforcement-learning exploration.
3.2. Real-world RL with VLAC
VLAC integrates structured action generation, dense progress evaluation, and PPO-based optimization in an asynchronous real-world reinforcement-learning system. Human-in-the-loop protocols address exploration failures and early training instability.
- Real-world deployment: Asynchronous execution lets robots upload observations and execute actions independently, reducing waiting time but creating possible observation-action temporal mismatch.Inference workers assign observations to idle VLA replicas within 0.1 seconds; action timestamps are adjusted during training to address lag.
- Reward and termination: During rollout, paired images and a task description are used to determine task completion and whether the second observation represents positive or negative progress.Separate prompts produce the done signal and signed relative-progress reward.
- PPO optimization: Clipping constrains policy updates, while entropy regularization encourages exploration and reduces premature convergence.The PPO objective uses a clipped surrogate loss and an entropy term weighted by a hyperparameter.
- Policy representation: VLAC generates language-based actions through a predefined structured template, recording selected-token logits for PPO policy-gradient updates.The template reduces autoregressive token requirements and preserves a parseable action format.
- PPO optimization: PPO combines action-token logits with a value estimate from a linear value head attached to VLAC hidden states.The value head is lightly pretrained on demonstration and early-exploration trajectories to reduce initial advantage variance.
- Human-in-the-loop: Three human-in-the-loop strategies—offline demonstration replay, return and explore, and human-guided exploration—target failed states, missing behaviors, and unstable early learning.The replay buffer uses human demonstrations with NLL updates, while operators reset difficult states or provide additional demonstrations.
4. Experiments
The experiments evaluate VLAC’s progress understanding, action generation, real-world reinforcement learning, human-assisted exploration, and multi-robot scaling across diverse settings.
- 4.4. VLAC Critic Performance: VLAC is evaluated on in-distribution and out-of-distribution datasets, including cross-entity, cross-viewpoint, human-hand, failure-process, and language-free scenarios.The evaluation covers Bridge, Droid, Dobb-E, RH20T, EgoDex, RoboFAC, and other datasets, with VOC, VROC, and Negative Rate used for progress understanding.
- 4.4. VLAC Critic Performance: Under one-shot conditions, VOC-F1 reaches 0.95 on RT1, while RoboNet reaches 0 in the zero-shot setting because it lacks language annotations and smooth temporal structure.Providing examples improves performance on new task processes and enables contextual reasoning across differing tasks, scenes, and robotic arms.
- 4.4. VLAC Critic Performance: VLAC distinguishes successful from failed processes and generalizes across heterogeneous entities, scenes, and tasks.Figure 5 includes zero-shot and one-shot examples, successful and failed processes, and a human manipulation dataset.
- 4.5. VLAC Actor Performance: VLAC action generation remains reasonable under lighting and scene changes, while removing task-progress pretraining substantially reduces success rate.Without progress understanding, the model may advance to placement before successfully grasping an object, despite producing reasonable actions.
- 4.6. Real-world RL Results: Human guided explore and Offline Demonstration Replay achieve 100% success rate across four tasks, whereas Return and explore performs only slightly better than Baseline.Offline Demonstration Replay learns faster than Human guided explore, and selecting effective return points is difficult.
- 4.7. Multi-Robot Scaling: With 8 robots, reaching 80% success requires 64 episodes per robot, compared with 147 for 4 robots and 325 for 2 robots.The results support reduced per-robot data requirements with more parallel robots, while the single-robot case deviates because its background is visually static.
5. Discussion and Conclusion
The work presents VLAC and a real-world RL framework that combines dense progress feedback with optional human intervention. It demonstrates strong manipulation improvements while identifying limits in intervention standardization, decoder generality, and multi-task stability.
- Contributions: VLAC converts pairwise multimodal observations and task descriptions into signed progress rewards, while unifying actor and critic functions in one architecture.The framework also supports progress-based completion detection and asynchronous real-world interaction.
- Results: 30% to 90% success within 200 real-world interaction episodes across four diverse manipulation tasks.The system also transfers under lighting and scene shifts without extra data collection and separates successful from failing trajectories.
- Limitations: Human intervention remains heuristic and operator-specific, limiting reproducibility and automation.Intervention timing, reset selection, and demonstration curation lack standardized quantitative triggers.
- Limitations: The PPO and tokenized delta–EEF interface does not directly generalize to diffusion, flow-matching, or continuous iterative action decoders.Those generators would require new abstractions for reward allocation, progress-to-score alignment, or Q-guided sampling.
- Limitations: Multi-task online RL remains unstable because of reward-scale drift, uneven negative signals, gradient interference, and episodic forgetting.The current system lacks task-adaptive normalization, uncertainty-weighted sampling, gradient-conflict mitigation, and related safeguards.
A. Evalauation tasks
The evaluation uses diverse real-world manipulation tasks spanning granular, rigid, and flexible objects and requiring distinct physical abilities. The tasks cover transfer, unfolding, precise placement, and force-sensitive sweeping.
- Task design: Five real-world manipulation tasks form a diverse kitchen-oriented evaluation spanning object types and manipulation abilities.The set includes rigid, flexible, and granular objects, as well as precise goal reaching and other skills.
- Task descriptions: Rice Scooping and Transfer requires scooping rice and pouring it into a cooker without spilling.Its difficulty comes from obtaining granular material and maintaining stability during transport.
- Task descriptions: Unfold Mat requires grabbing, lifting, and releasing a folded mat so it spreads properly on the table.Success depends on selecting a suitable grasp location and lifting high enough for natural spreading.
- Task descriptions: Pick and Place Bowl requires gently gripping a bowl rim and placing the bowl precisely at the plate center.The task is successful when the bowl is properly placed on the plate.
- Task descriptions: Desktop Sweep Disposal requires pushing table trash into a nearby can with appropriate reach and force.Excessive or insufficient force can cause failure.
B. Dataset
The dataset mixture combines conversational, robotic, spatial, and pairwise image-difference data. These sources support multimodal understanding, spatial reasoning, and fine-grained progress comparison.
- Dataset mixture: LLAVA contributes multi-turn dialogue and visual question answering data for general multimodal and conversational abilities.
- Dataset mixture: RoboVQA provides visual question answering data from robotic tasks to strengthen multimodal understanding in robotics.
- Dataset mixture: SpatialVQA supplies RGB-image spatial reasoning data covering depth estimation, object detection, and related capabilities.
- Dataset mixture: Spot-the-diff and InstructPix2Pix provide pairwise image-difference comparisons for fine-grained progress understanding.
- Dataset mixture: Table 4 presents an overview of the data mixture.
C. Scene Transfer
Figure 9 illustrates scene and lighting transfer used in the evaluation. It provides the visual context for assessing transfer across changed environments.
- Scene Transfer: Scene and lighting transfer are illustrated as evaluation conditions.
D. Multi-Robots Scaling
The section examines multi-robot real-world RL, where environmental and robot-status differences produce divergent learning outcomes under a shared policy. It also presents experiments varying the number of robots trained simultaneously on Pick and Place Bowl.
- Multi-robot learning imbalance: Background clutter, lighting, and robot status differences create divergent per-robot success curves under a shared policy.The passage identifies these differences as small covariate shifts.
- Multi-robot learning imbalance: Robots with higher success rates tend to learn faster, whereas lower-success robots show little improvement over time.
- Experimental results: Figure 10 demonstrates Pick and Place Bowl results with different numbers of robots trained simultaneously.
E. Author contributions
The author-contributions section records collective writing participation and lists individual contributions spanning robot and data engineering, infrastructure, VLAC pretraining, design, experiments, and analysis.
- Author contributions: All authors contributed to writing.
- Author contributions: Qi Zhang contributed to robot and data engineering, architecture and algorithm design, experiment analysis, paper writing, and VLAC pretraining.
- Author contributions: Tianyi Zhang contributed to experiment analysis, robot engineering, infrastructure engineering, and VLAC pretraining.