PaperlayerView exact revision on arXiv

Source-linked AI summary

Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation

Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Yuheng Ji, Cheng Chi, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, Shaoxuan Xie, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang

arXiv:2512.23703v1cs.RO

TL;DR

Real-world robotic reinforcement learning is limited by unreliable reward design, including process models that lack step awareness and robust perception and shaping methods that can misguide optimization. Robo-Dopamine introduces multi-view, step-aware GRM rewards and policy-invariant shaping, achieving strong reward assessment and raising near-zero policies to 95% success in about 150 online rollouts after one-shot adaptation.

  • Problem

    Reward design limits real-world robotic reinforcement learning because existing process reward models can miss fine-grained progress and dense-reward shaping can induce a semantic trap.

  • Method

    Dopamine-Reward trains a general-purpose, step-aware GRM from multi-view data using step-wise progress discretization and multi-perspective reward fusion, while Dopamine-RL applies policy-invariant reward shaping.

  • Results

    GRM achieves over 92.8% progress-assessment accuracy and 0.953 Value-Order Consistency, while one-shot-adapted Dopamine-RL reaches 95% success from near-zero within approximately 150 online rollouts.

  • Takeaways & Limitations

    The framework supports dense, step-aware reward guidance for efficient policy improvement while preserving the optimal policy and avoiding the semantic trap.

  • Takeaways & Limitations

    The method's progress-consistency predictions can diverge substantially in out-of-distribution observations, while remaining consistent in familiar states.

Abstract

from arXiv · show

The primary obstacle for applying reinforcement learning (RL) to real-world robotics is the design of effective reward functions. While recently learning-based Process Reward Models (PRMs) are a promising direction, they are often hindered by two fundamental limitations: their reward models lack step-aware understanding and rely on single-view perception, leading to unreliable assessments of fine-grained manipulation progress; and their reward shaping procedures are theoretically unsound, often inducing a semantic trap that misguides policy optimization. To address these, we introduce Dopamine-Reward, a novel reward modeling method for learning a general-purpose, step-aware process reward model from multi-view inputs. At its core is our General Reward Model (GRM), trained on a vast 3,400+ hour dataset, which leverages Step-wise Reward Discretization for structural understanding and Multi-Perspective Reward Fusion to overcome perceptual limitations. Building upon Dopamine-Reward, we propose Dopamine-RL, a robust policy learning framework that employs a theoretically-sound Policy-Invariant Reward Shaping method, which enables the agent to leverage dense rewards for efficient self-improvement without altering the optimal policy, thereby fundamentally avoiding the semantic trap. Extensive experiments across diverse simulated and real-world tasks validate our approach. GRM achieves state-of-the-art accuracy in reward assessment, and Dopamine-RL built on GRM significantly improves policy learning efficiency. For instance, after GRM is adapted to a new task in a one-shot manner from a single expert trajectory, the resulting reward model enables Dopamine-RL to improve the policy from near-zero to 95% success with only 150 online rollouts (approximately 1 hour of real robot interaction), while retaining strong generalization across tasks. Project website: https://robo-dopamine.github.io

1. Introduction

The paper identifies reward design as the central obstacle to real-world robotic reinforcement learning and introduces Dopamine-Reward and Dopamine-RL to address reward assessment and shaping. Its methods combine multi-view, step-aware reward modeling with policy-invariant shaping, achieving strong reward accuracy and efficient policy improvement.

  • Motivation: The paper frames effective reward design as the primary obstacle to applying reinforcement learning in real-world robotics.Sparse rewards hinder exploration, handcrafted dense rewards limit scalability, and existing process reward models have perceptual and structural weaknesses.
  • Proposed approach: Dopamine-Reward learns general-purpose, step-aware process rewards from multi-view inputs using GRM construction and multi-perspective reward fusion.Its annotation pipeline covers over 3,400 hours, 100K trajectories, and more than 350 manipulation tasks.
  • Empirical validation: Dopamine-RL improves policies from near-zero to 95% success within approximately 150 online rollouts after one-shot GRM adaptation, with some tasks reaching 100%.The interaction budget is approximately one hour of real robot interaction.
  • Proposed approach: Dopamine-RL applies theoretically sound policy-invariant reward shaping so dense GRM rewards support self-improvement without changing the optimal policy.This design is intended to avoid the semantic trap caused by unsound dense-reward shaping.
  • Empirical validation: GRM achieves over 92.8% progress-assessment accuracy and a 0.953 Value-Order Consistency score, outperforming established baselines.These results are reported across experiments on more than 10 simulated and 8 real-world tasks.

2. Related Work

Related work positions reinforcement learning as a route to robust robotic skills and describes learned process reward models as a way to densify feedback. It highlights unresolved concerns about generalization, reward allocation, and perception in existing approaches.

  • Reinforcement Learning for Robotic Skills: Reinforcement learning research for robotic skills spans offline, online, and mixed policy-optimization strategies while targeting complex, contact-rich, and dexterous tasks.The related work describes RL as enabling novel and robust strategies beyond imitation learning.
  • Learned Process Reward Models: Learned process reward models provide denser feedback than binary outcome rewards by predicting progress between observations or assigning per-frame progress scores.These methods are motivated by the difficulty of exploration in complex, long-horizon tasks under sparse outcome rewards.
  • Learned Process Reward Models: Existing process reward models remain limited by task-specific designs, nearly uniform reward allocations, and single-view perception that can miss fine-grained progress.These limitations affect generalization, sub-step salience, and observation of manipulation scenes with occlusions.

3. Method

The method combines a step-aware, multi-view General Reward Model with multi-perspective fusion and policy-invariant shaping for robust robotic reinforcement learning. Its design discretizes progress structurally, reduces perceptual and prediction drift, filters uncertain estimates, and preserves the optimal action under shaping.

  • Overview: The framework combines Dopamine-Reward, which learns step-aware rewards from multi-view inputs, with Dopamine-RL, which addresses theoretical flaws in conventional reward shaping.The two components jointly target reward-modeling limitations and policy-optimization misalignment.
  • General Reward Model Construction: Step-wise discretization segments expert trajectories into synchronized multi-view states and defines normalized global progress across the resulting sequence.Progress is represented as Φ(si) = i/M after adaptive sampling within human-annotated sub-task segments.
  • General Reward Model Construction: Hop-based supervision normalizes forward and regressive transitions, keeping reconstructed progress within [0, 1] while reducing iterative error accumulation.Forward changes are scaled by remaining distance to the goal, whereas regression is scaled by progress already covered from the initial state.
  • Multi-Perspective Progress Fusion: Multi-perspective fusion combines incremental, forward-anchored, and backward-anchored predictions to balance local precision, initial stability, and goal sensitivity.The resulting fused signal is described as more accurate and resistant to drift for subsequent reward shaping.
  • Progress Consistency Checking: Consistency-aware weighting and conservative updates suppress uncertain out-of-distribution progress estimates, reducing opportunities for reward hacking.The update retains the previous progress estimate when confidence approaches zero and trusts the new estimate when consistency is high.
  • Policy-Invariant Reward Shaping: Policy-invariant shaping preserves the optimal action because the shaped value differs from the original by a state-dependent shift identical across actions.The cumulative shaping term telescopes to a boundary term depending only on the initial state, unlike naive dense-reward use that can encourage stagnation.

4. Experiments

Experiments evaluate GRM’s progress perception and Dopamine-RL’s policy performance, efficiency, generalization, and design choices across simulated and real-world manipulation tasks.

  • 4.1.1. Video Frame Rank-Correlation: Ours-8B (Multi-View) sets a new state-of-the-art across all VOC benchmarks and temporal sampling densities.The evaluation uses sparse, medium, and dense temporal sampling across diverse robotics and manipulation datasets.
  • 4.1.2. Task Completion Judgment: 92.8% versus 83.9% shows that Multi-View GRM substantially improves task-completion judgment over its Single-View variant.Alternative viewpoints help verify progress when one camera is occluded.
  • 4.2. Performance, Efficiency, Generalization (RQ2): Dopamine-RL significantly outperforms all baselines in final success rate and sample efficiency with fewer environment interactions.Table 3 measures sample efficiency by episodes needed to reach 80% of final success rate.
  • 4.2. Performance, Efficiency, Generalization (RQ2): OOD performance drops 8–20% for Dopamine-RL versus 50–60% for BC under object, layout, and background changes.The comparison indicates stronger retention of performance under distribution shifts.
  • 4.3. Ablation Studies (RQ3 & RQ4): Removing Multi-Perspective Progress Fusion causes 15.0%, 19.3%, and 22.5% absolute performance drops across single-estimator variants.The incremental-only variant is especially vulnerable to error drift over long horizons.
  • 4.3. Ablation Studies (RQ3 & RQ4): Removing policy-invariant reward shaping causes a 43.7% performance drop as agents reach “good-enough” states and stagnate.Zero-shot GRM errors on OOD corner cases additionally cause a 21.8% success-rate drop.

5. Conclusion

Robo-Dopamine addresses reward design for real-world robotics with a multi-view, step-aware reward model and policy-invariant reward shaping. Experiments report strong reward accuracy, efficient policy improvement, and generalization.

  • 5. Conclusion: Robo-Dopamine combines a 3,400-hour multi-view GRM with Policy-Invariant Reward Shaping for dense guidance without altering the optimal policy.The framework targets occlusion-related perceptual limits and the semantic trap in reward shaping.
  • 5. Conclusion: Policies improve from nearly-zero to ∼95% success in ∼150 interaction rollouts while exhibiting strong generalization.The conclusion presents this as evidence of sample-efficient policy learning across diverse tasks.

Robotic Manipulation

The supplementary material gives additional method, theory, dataset, and experimental details for Dopamine-Reward and Dopamine-RL.

  • Robotic Manipulation: Supplementary sections provide proofs, dataset composition and sampling details, method elaboration, and additional experimental results.The material includes proofs concerning bounded global progress, the semantic trap, and optimal-policy invariance.

A.1. Proof of Bounded Global Progress (Proof 1)

The proof establishes that recursively reconstructed global progress remains bounded in [0, 1]. It also connects the shaping formulation to discounted continuous-time dynamics and policy-preserving boundary terms.

  • Bounded global progress: Φ⋆(s_t) remains in [0, 1] for all steps when Φ⋆(s_0)=0 and predicted hops lie in [−1, 1].The induction handles positive and negative hops separately under the recursive update rule.
  • Inductive step: Positive hops preserve the bound because the update can be expressed as a convex combination of the current progress and the hop value.
  • Inductive step: Negative hops preserve nonnegativity because the update becomes Φ⋆(s_t)=G(1+H), with G≥0 and H∈[−1,0).
  • Discounting: Time consistency yields exponential discounting, with discrete factor γ=e^−λh for control interval h.
  • Continuous-time consistency: The policy-invariant shaping term is consistent with discounted continuous-time dynamics, whose cumulative integral reduces to an initial-state boundary term.The discrete term is interpreted as a first-order Euler discretization, and its continuous cumulative form depends only on the initial state.

A.5. Policy Invariance under GRM (Proof 5)

The proof shows that GRM shaping preserves the optimal policy because discounted shaping rewards telescope to a trajectory-independent boundary term. Consequently, shaped Q-values shift by a state-dependent constant without changing action order.

  • The cumulative shaping reward telescopes to a boundary term independent of future actions, preserving policy invariance.
  • With r_GRM=r_env+F and F(s_t,s_{t+1})=γΦ(s_{t+1})−Φ(s_t), intermediate discounted terms cancel across the finite-horizon sum.
  • Under Φ∈[0,1] and γ<1, the terminal term vanishes as T→∞, leaving −Φ(s) regardless of the future trajectory.
  • Because shaping shifts Q-values by a state-dependent constant, the ordering of actions is preserved.

B.1.1. Real-World Datasets

The GRM training corpus combines real-world robotics, simulation, and human manipulation data to support broad embodiment and task generalization. Its sampling and labeling procedures emphasize diverse progress dynamics, including regression and zero-progress transitions.

  • Real-world sources: Real-world datasets contribute visual and embodiment diversity, helping GRM remain stable across lighting, textures, and kinematic structures.
  • Corpus composition: The final corpus contains 35 million samples expanded from raw sources through multi-view augmentation, with real-world data forming the majority.The corpus combines approximately 3,400 hours of footage from robotics, simulation, and human-centric domains.
  • Task diversity: The dataset spans atomic manipulation primitives and complex multi-stage tasks, producing a long-tail distribution of task categories.
  • Progress labeling: Hop labels normalize progress relative to the remaining or accumulated task distance, making late-stage progress and regression statistically meaningful.
  • Sampling and balancing: Stratified sampling balances hop magnitudes and temporal gaps, while explicit zero-hop samples reduce bias toward static or near-static segments.

C.1. GRM Training and Evaluation

GRM training uses large-scale multimodal infrastructure and evaluates both temporal progress ordering and task-completion judgment. The resulting GRM-based pipelines achieve stable LIBERO-Goal learning in two RL settings and support real-world adaptation with ConRFT.

  • Training setup: The GRM is trained with RoboBrain 2.0 at 3B and 8B parameter scales on a 35M-sample corpus using distributed H100 infrastructure.
  • Evaluation protocol: GRM evaluation measures temporal progress with VOC and task completion using 60 real-world rollouts across three challenging tasks.The rollouts are balanced among successful, partially successful, and failed episodes.
  • Simulation setup: A PPO plus OpenVLA-OFT pipeline uses clipped updates, action-level value estimation, and partial reset to improve sample efficiency for chunked policies.
  • Simulation experiments: 81% average success is achieved across LIBERO-Goal tasks using ReinFlow with π0 and GRM-based reward shaping.The experiment uses 10 long-horizon manipulation tasks and approximately 50 hours of training.
  • Real-world setup: Real-world experiments use ConRFT, an offline-to-online method combining behavior cloning, calibrated Q-learning, and consistency-policy adaptation.

C.3.2. Real-World Tasks

The real-world evaluation spans eight representative robotic manipulation tasks, covering single-arm, dual-arm, fine-grained, and long-horizon settings.

  • Eight real-world tasks cover single-arm, dual-arm, fine-grained, and long-horizon manipulation settings.The suite includes Zip the Bag, Insert Square, Pick and Place, Complete Circuit, Arrange Flowers, Fold Towel, Build Blocks, and Cap the Pen.
  • Insert Square requires millimeter-level position and orientation adjustment to place a four-hole block onto matching pegs.
  • Pick and Place combines randomized initial positions with sequential grasping and placement, making it a single-arm fine-grained long-horizon task.
  • Complete Circuit requires coordinated dual-arm sequencing: battery insertion followed by switch activation to light the bulb.
  • The remaining tasks test coordinated fine manipulation, including zipping a bag, arranging flowers, folding a towel, stacking blocks, and capping a pen.

D. More Qualitative Results

Qualitative results show that GRM generalizes progress estimation across tasks, remains stable across sampling intervals, and supports recovery from real-world disturbances.

  • GRM Predictions on Diverse Tasks: GRM captures monotonic progress in successful trajectories and identifies stagnation or regression in failed attempts across diverse tasks.
  • Robustness to Temporal Intervals: Progress curves remain consistent when frame intervals vary from 10 to 100 frames, indicating robustness to temporal granularity.
  • Real-World RL Rollout Visualization: After a human moves the target slot, GRM sharply lowers estimated Progress, and the policy recovers alignment before completing insertion.The rollout policy was trained for approximately 20 minutes and achieved over 95% success.

E. Future Work

Future work targets faster GRM inference, continuous video reasoning, broader task generalization, and multimodal reward signals.

  • Efficient GRM Inference: The current VLM-based GRM is accurate but has high computational latency that can bottleneck online RL training.
  • Continuous Video Stream Reasoning: Future GRM versions will reason over continuous video and historical context to capture motion trends and resolve ambiguities in static frames.
  • Large-Scale Generalization: The framework will be tested on highly dynamic tasks and long-horizon mobile manipulation to assess the limits of policy-invariant reward shaping.
  • Multi-Modal Reward Modeling: Tactile and auditory inputs are planned to complement vision for contact-rich operations and discrete events such as latch clicks.
Loading 2512.23703v1…