PaperlayerView exact revision on arXiv

Source-linked AI summary

Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation

Yijie Xu, Haopeng Jin, Run Zhou, Shengbang Liu, Sixiang Chen, Hongyang Cheng, Sicheng Hu, Peterson Co, Jinwen Luo, Huajie Tan, Shanghang Zhang

arXiv:2608.15680v1cs.ROcs.AI

TL;DR

Robotic manipulation needs dense, reliable rewards that remain informative under temporal ambiguity and out-of-distribution execution. Robo-Dopamine 2.0 addresses this with history-conditioned pairwise rewards, signed progress modeling, and curriculum training, improving average VOC from 0.967 to 0.986 while supporting gains across offline, simulation, and real-world settings.

  • Problem

    Existing visual reward models have limited evidence for evaluating manipulation progress under temporal ambiguity and out-of-distribution execution, where static endpoint pairs may be insufficient.

  • Method

    Robo-Dopamine 2.0 combines history-conditioned pairwise rewards, an OOD-aware signed progress space, and Signed-Hop curriculum training with transition-aware replay.

  • Results

    The framework shows complementary gains across offline ordering, simulation, and real-world manipulation settings, while panels raise average VOC from 0.967 to 0.986.

  • Takeaways & Limitations

    The shared pairwise interface supports both offline progress estimation and closed-loop reward shaping across the evaluated manipulation settings.

  • Takeaways & Limitations

    Online reward queries are restricted to observations available before or at the current rollout step, excluding future rollout observations.

Abstract

from arXiv · show

Vision-language-action (VLA) models improve robotic manipulation but remain vulnerable to compounding errors, scene changes, and off-trajectory states. Reinforcement learning can refine pretrained VLA policies, yet sparse success signals hinder exploration, while engineered dense rewards are costly and task-specific. Existing learned visual reward models often rely on static before-after observations, causing temporal ambiguity and weak discrimination between robustness-preserving variations and task-invalid failures under out-of-distribution (OOD) execution. We introduce Robo-Dopamine 2.0, a history- and OOD-aware process reward model with a pairwise prediction interface. It combines (1) history-conditioned pairwise rewards that use source-aligned reference panels for synthetic OOD queries and observed rollout history for online queries, while preserving the queried endpoints, and (2) an OOD-aware signed progress space that represents valid progress, robustness, failure, and recovery. A Signed-Hop Curriculum with transition-aware replay learns coarse execution ordering before fine-grained progress calibration. We also construct an OOD trajectory dataset and a five-family benchmark. Reference panels improve mean visual order consistency (VOC) from 0.967 to 0.986 and OOD-robust VOC from 0.906 to 0.958. With the same 400K pairwise-reward budget, Signed-Hop training with 25% replay reaches 0.9872 mean VOC, compared with 0.9858 for a matched-pool shuffled control. In downstream reinforcement learning, the full model achieves 86.8% mean RoboTwin success and 71/80 successful real-world insertions.

Introduction

Robo-Dopamine 2.0 addresses the difficulty of evaluating progress in long-horizon, contact-rich manipulation by combining history-conditioned pairwise rewards with an OOD-aware signed progress space. A Signed-Hop Curriculum with transition-aware replay further learns coarse execution ordering before fine-grained progress calibration.

  • Motivation: Long-horizon, contact-rich manipulation remains difficult because compounding errors, unseen scene changes, and off-trajectory states require recovery.Reinforcement learning can improve policies through interaction, but practical real-robot deployment remains challenging.
  • Problem: Existing visual and language-conditioned reward models provide dense feedback, yet remain limited in accurately evaluating execution progress under challenging manipulation scenarios.Robot-specific methods add stage- or process-aware progress estimation, but the evaluation limitation persists.
  • Method: Robo-Dopamine 2.0 combines history-conditioned pairwise rewards with an OOD-aware signed progress space for valid progress, robustness, failure, and recovery.It uses expert or rollout history for standard and online queries, and phase-aligned successful-reference panels for synthetic OOD queries without replacing edited endpoints.
  • Training: The Signed-Hop Curriculum first learns coarse ordering among successful, robust, and failed states, then refines fine-grained progress differences using transition-aware replay.Replay preserves informative large-transition pairs to improve progress calibration across diverse execution scenarios.

Related Work

Prior work establishes generalist robot policies, visual reward learning, and temporal/process-aware reward modeling. Robo-Dopamine 2.0 addresses gaps in endpoint-oriented reward methods by combining query-dependent temporal context with explicit OOD failure and recovery supervision.

  • Generalist Robot Policies: Generalist robot policies and VLAs scale visuomotor learning and semantic generalization across tasks and embodiments, with RT-1, RT-2, and OpenVLA providing representative foundations.RT-1 studies large-scale real-world robot control, RT-2 transfers web-scale semantic knowledge into action prediction, and OpenVLA supports adaptation to new manipulation domains.
  • Visual Reward Learning: Visual reward learning spans preference supervision, language-conditioned state changes, visual or video-language feedback, pairwise preferences, general-purpose rewards, failure-aware feedback, and stage-aware rewards.Existing methods remain largely endpoint-oriented or success-centric and do not combine query-dependent temporal context with explicit OOD failure/recovery supervision.
  • Temporal and Process-aware Reward Modeling: Temporal context has been explored in recurrent RL, history-aware manipulation policies, and order-sensitive proxy rewards, primarily through policy memory or trajectory matching.Robo-Dopamine 2.0 instead targets temporal ambiguity within pairwise reward estimation.
  • Temporal and Process-aware Reward Modeling: Robotic process reward models investigate multi-view relative progress, while Robo-Dopamine 2.0 extends process-aware reward modeling toward temporal ambiguity in pairwise estimation.The supplied passage introduces this process-reward direction but is truncated before describing its full prior-work scope.

Method

Robo-Dopamine 2.0 preserves a pairwise reward interface while conditioning predictions on ordered temporal context and source-aligned or rollout-history panels. Its signed OOD supervision and Signed-Hop curriculum model progress, robustness, failure, and recovery for closed-loop reinforcement learning.

  • History-conditioned pairwise rewards: The GRM predicts signed relative progress from an instruction, task-reference anchors, ordered context, and queried endpoint states.The static baseline removes temporal context, while the endpoints remain the actual queried observations.
  • History-conditioned pairwise rewards: Synthetic OOD queries use source-only panels while preserving edited endpoints; standard queries use same-episode history, and online queries use observed rollout history.Online panels contain only observations available before or at the current rollout step, preventing future-information leakage.
  • Signed OOD supervision: Signed supervision separates valid execution from task-invalid failure, assigns failure depth by aligned source phase, and extends Hop normalization to failure and recovery.Negative states receive negative coordinates based on phase-aligned depth rather than calibrated failure probabilities.
  • Signed OOD supervision: Training covers valid, failure, recovery, and within-failure pairs, while same-index robust pairs enforce invariance and fixed quotas prevent positive pairs from dominating.Robust operators include occlusion, background variation, distractors, and non-target color changes; negative operators include wrong-object interaction and empty grasp.
  • Signed-Hop curriculum: The Signed-Hop curriculum trains two 200K-example stages totaling 400K pairs, learning global geometry before local calibration with replay of large-Hop examples.Examples are ranked by |Hba| within pair families; Stage 1 samples large-Hop examples, while Stage 2 emphasizes fine and zero-Hop calibration.

Experiments and Results

Experiments show that history-conditioned panels and OOD supervision improve reward-model consistency, while Signed-Hop ordering adds a smaller curriculum benefit. These gains transfer to reinforcement learning, reaching 86.8% mean RoboTwin success and 71/80 successful real-world insertions.

  • Reward-model evaluation: 0.986 average VOC and 0.958 OOD-robust VOC result from adding reference panels, up from 0.967 and 0.906, respectively.The same-checkpoint comparison isolates training-aligned temporal context from model capacity and training data.
  • Reward-model evaluation: 96.1% task-completion accuracy is achieved by Robo-Dopamine 2.0-8B, exceeding GPT-5.5 at 81.1% and the other listed vision-language models.The compared models are Claude-Opus-4.8, Gemini-3-Pro, Qwen3.5-397B-A17B-fp8, and RoboBrain 2.0.
  • Ablations: 0.987 panel VOC is reached with Robo-Dopamine 2.0 using 3×3 panels, versus 0.974 with OOD no-panel training; panel gains exceed those from OOD supervision alone.Across backbones, one or two between-state panels perform best, while runtime construction introduces distribution changes and a visual-token trade-off.
  • Signed-Hop curriculum: 0.9872 average VOC with 25% replay exceeds the matched-pool shuffled control at 0.9858 and no replay at 0.9866 under the fixed 400K budget.Signed-Hop learns global geometry from large-Hop pairs before refining local calibration with small and zero-Hop pairs; gains appear in every benchmark family, especially OOD cases.
  • Downstream reinforcement learning: 86.8% mean RoboTwin success is achieved when rollout-history panels and OOD supervision are combined, versus 72.2% with neither and 85.8% for the 1,000-epoch sparse baseline.Within 300 epochs, history panels raise mean success to 75.1%, OOD supervision to 80.6%, and their combination improves the mean by 6.2 and 11.7 points over those conditions.
  • Real-world evaluation: 71/80 successful insertions and 15/20 full-sequence successes are achieved by C11, versus 48/80 and 8/20 for Event Sparse.Rollout-history panels provide the largest gain, with C01 and C11 each adding 21 successful insertions and 6 full-sequence successes relative to their corresponding static-panel conditions.

Conclusion

Robo-Dopamine 2.0 unifies history-conditioned pairwise prediction, OOD-aware signed progress modeling, and Signed-Hop curriculum training in one interface. Results indicate complementary gains across offline ordering, simulation, and real-world manipulation, while supporting both progress estimation and closed-loop reward shaping.

  • Method: Robo-Dopamine 2.0 combines history-conditioned pairwise prediction, an OOD-aware signed progress space, and a Signed-Hop curriculum in one reward interface.The signed space represents valid progress, robustness-preserving variations, task-invalid failures, and recovery.
  • Method: Standard queries use same-episode expert context, synthetic OOD queries retain edited endpoints with aligned successful-reference panels, and online queries use observed rollout history.These query designs provide temporal context while preserving the queried endpoints for synthetic OOD cases.
  • Results: Complementary gains from temporal context, OOD supervision, curriculum ordering, and replay appear across offline ordering, simulation, and real-world manipulation settings.The same pairwise interface supports offline progress estimation and closed-loop reward shaping.
  • Future work: Future work will broaden task, OOD, and real-world evaluation.The conclusion identifies broader evaluation as the next direction.

Supplementary Material … Structured Prompt Design

The supplementary pipeline constructs source-aligned synthetic OOD trajectories, parses tasks into manipulation primitives, localizes edits to valid execution phases, and uses temporally ordered multi-reference prompts. Structured prompts preserve task semantics, camera-specific evidence, physical and visual invariants, and single-image output constraints.

  • OOD Data Construction Pipeline: Synthetic OOD data combines successful RoboCasa, LIBERO, and RoboTwin 2.0 trajectories with real-robot AgiBot World data, using task- and phase-aware edits.Negative edits alter task-relevant execution conditions, whereas robust edits change nuisance appearance while preserving the intended task state.
  • OOD Data Construction Pipeline: The six-step construction pipeline normalizes instructions, assigns compatible operators, localizes valid phases, generates constrained multi-reference edits, and retains provenance for reward-pair construction.Provenance includes the source episode, frame, camera, prompt, and references.
  • Task Parsing and Operator Assignment: Task parsing maps instructions to seven primitives—grasp–lift, place–transfer, open, close, toggle-on, toggle-off, and push–move—and extracts target phrases to bind edits to task objects.Semantic guards avoid ambiguous identity changes among multiple legitimate targets and restrict non-target color changes when target color is specified.
  • Task Parsing and Operator Assignment: Primitive-specific operator mappings distinguish failure edits such as wrong object, empty grasp, non-release, and target replacement from robust nuisance edits such as occlusion, distractors, and non-target color changes.The allowed operators depend on whether the task is grasping, placing, opening, closing, toggling, or pushing.
  • Phase-Aware Temporal Localization: Valid progress windows are [0.18, 0.80] for grasp–lift, [0.28, 0.95] for place–transfer, [0.15, 0.85] for open and close, [0.10, 0.75] for toggles, and [0.12, 0.85] for push–move.Composite trajectories switch active primitives at progress values 0.45, 0.65, and 0.35 depending on the task sequence.
  • Multi-Reference Edit Construction: Each queried frame uses up to 13 chronologically ordered references from the same successful episode and camera, while the queried frame remains the only editable image.This preserves camera projection, visible surfaces, and gripper geometry while supplying evidence about target identity, interaction, transport, and final state.
  • Structured Prompt Design: The structured prompt separates input, scene and target binding, primitive, operator, physical, view-specific, and output contracts.It identifies read-only references, the editable final image, task semantics, the OOD branch, camera and robot invariants, and the required single-image output.
  • Structured Prompt Design: Prompts preserve queried resolution 320×240, camera and image characteristics, robot configuration, contact and occlusion geometry, illumination, and local scene consistency while prohibiting unnecessary restyling.Head-camera requests emphasize global scene consistency, wrist-camera requests constrain edits around the gripper and target, and the editor must return exactly one edited image.

Negative Counterfactual Operators … Generation and Provenance

The paper constructs OOD observations by applying task-invalidating or robustness-preserving edits while fixing source pose, phase, and semantics as appropriate. Camera-specific constraints and detailed provenance preserve alignment between edited queries and successful source trajectories.

  • Negative Counterfactual Operators: Negative operators alter task-relevant execution conditions while keeping the source robot pose and trajectory phase fixed.They include wrong-object, empty-grasp, non-release, and target-replacement edits.
  • Negative Counterfactual Operators: Wrong-object, empty-grasp, non-release, and target-replacement edits preserve selected scene geometry while introducing task-invalid interaction or identity conditions.Examples include replacing the target or held content, clearing the grasp path, and converting release into continued holding.
  • Robustness-Preserving Operators: Robustness operators change image appearance without changing task semantics and retain the same progress value as their aligned source states.The operators cover foreground occlusion, background changes, distractor addition or removal, and non-target color changes.
  • Camera-Specific Geometric Constraints: Camera-specific prompt blocks adapt each semantic edit to the distinct geometry and visibility constraints of head and wrist views.Head prompts prioritize global layout and projected grasp corridors, whereas wrist prompts restrict edits to local gripper–target regions.
  • Camera-Specific Geometric Constraints: Edits preserve source depth ordering and robot occlusions so different camera projections express consistent trajectory-level OOD semantics.Inserted or replaced objects remain behind visible metal finger edges, and occluding robot geometry is not removed.
  • Generation and Provenance: Each frame–camera edit is generated in batch from ordered reference images, the queried image, and a fully instantiated prompt using Gemini 3.1 Flash Image.Every request receives a unique identifier tied to source dataset, task, episode, frame, camera, operator, OOD metadata, prompt, backend, timestamp, and source path.
  • Generation and Provenance: Source-only temporal context reconstructs successful reference observations at the same progress index while leaving edited before–after evidence exclusively in the OOD query.Recorded provenance separately tracks source state, edited observation, camera projection, and temporal context for signed-progress compilation.

Computing Infrastructure and Training Cost

Training runs use substantially different accelerator allocations: reward-model fine-tuning and RoboTwin simulation use eight H100 GPUs, while real-world ConRFT training uses one RTX 4090. The retained records report accelerator allocations only because host-system and CUDA details are unverified.

  • Accelerator allocation: Reward-model fine-tuning and RoboTwin simulation policy learning use eight NVIDIA H100 GPUs with 80 GB per GPU, totaling 640 GB of accelerator memory.This allocation applies to every training run.
  • Accelerator allocation: Real-world ConRFT training uses one NVIDIA RTX 4090 instead of the eight-GPU H100 configuration.
  • Reporting scope: Verified records omit host CPU, system RAM, operating-system, and CUDA details, so only accelerator allocations are reported.The paper avoids inferring unverified model or version information.

Simulation Reinforcement Learning Details

Simulation reinforcement learning optimizes task-specific OpenVLA-OFT SFT checkpoints with GRPO while keeping the GRM frozen and conditioning history-enabled panels on rollout prefixes. Policies use camera and proprioceptive observations with chunked actions, distributed rollouts, and task-specific training schedules and horizons.

  • GRPO optimizes task-specific OpenVLA-OFT SFT checkpoints in RLinf across RoboTwin 2.0 tasks.
  • The frozen GRM constructs history-enabled panels from observed rollout prefixes, while policies receive head-camera RGB and 14-D proprioception.Policies predict 14-D actions in 25-action chunks and are LoRA-tuned in BF16 with FSDP.
  • GRPO uses 128 parallel environments, groups of eight rollouts, normalized group-relative advantages, and γ = 1.
  • Training runs 1,000 epochs for the sparse baseline and 300 epochs for C00–C11, with deterministic 128-episode evaluations every 20 epochs.Episode horizons are 200, 150, and 400 steps, and learning rates are 10−4, 2×10−4, and 2×10−4 for place_empty_cup, place_container_plate, and handover_block, respectively.

Real-World Experiments

Real-world experiments evaluate history-aware reward shaping for repeated, precision-sensitive square insertion using a frozen reward model and ConRFT policy optimization. Conditions share the same sparse task reward, initialization, demonstrations, observations, resets, interventions, and evaluation, while GRM-shaped variants add reward guidance.

  • Experimental setup: The physical workcell uses two Franka manipulators, a Pika teleoperation interface, and calibrated ZED cameras providing synchronized wrist and third-person RGB observations.The GRM remains frozen during online policy learning, and history-enabled conditions construct temporal panels from observed rollout prefixes without online source alignment or repetition labels.
  • Experimental setup: Each trial performs K = 4 consecutive back-and-forth insertion attempts, requiring millimeter-scale alignment of a four-hole square block onto four upright pegs.Repeated pickup, alignment, and insertion observations expose temporal aliasing beyond one-shot manipulation accuracy.
  • Policy optimization: ConRFT optimizes the policy from 40 teleoperated trajectories using behavior cloning and calibrated Q-learning with a conservatively regularized critic.The policy retains a frozen Octo-Small observation backbone and replaces the original action head with a consistency policy.
  • Policy optimization: During online adaptation, autonomous transitions enter replay buffer R, unsafe actions trigger human intervention, and corrected trajectories return to demonstration buffer D.Updates sample equally from D and R; online critic updates remove the offline conservative penalty while shifting policy-term weights toward reward-driven improvement.
  • Comparison protocol: Every condition uses an event-sparse reward after each successful insertion, while C00–C11 additionally use GRM shaping and the terminal-only four-attempt pilot is excluded after failing to learn.Reported variants share policy initialization, demonstration data, observations, resets, interventions, and evaluation; the experiment uses 20 trials, counting the latter outcome only when all four attempts succeed.

Additional Ablation Details

Additional ablations show that moderate replay, history-conditioned panels, OOD supervision, and structured 3×3 layouts improve reward-model calibration and robustness. Benefits diminish with excessive replay or visual-panel capacity.

  • Replay-rate sensitivity: 0.98724 average VOC occurs at ρ = 0.25, compared with 0.9866 without replay and 0.98668 at ρ = 0.60.The selected replay rate maximizes overall average VOC, while ρ = 0.30 remains nearly unchanged at 0.98722.
  • Replay-rate sensitivity: 0.9948 OOD-negative VOC is reached at ρ = 0.60, while OOD-robust and OOD-temporal VOC peak near moderate replay and decline at larger ratios.Thus, ρ = 0.25 achieves the highest observed overall average rather than the best score on every branch.
  • Trajectory-level progress curves: The panel-conditioned model follows reference ordering most closely during early and middle foreground-occlusion phases, but retains a late-episode gap.Static GRM stays near zero mid-episode, while history does not eliminate sensitivity to strong foreground edits.
  • Trajectory-level progress curves: All three learned curves preserve episode order and reach completion on the in-distribution drawer case, although the panel-conditioned curve saturates early at final states.This retained behavior exposes temporal-context benefits alongside remaining endpoint-calibration error.
  • Configuration ablations: OOD supervision supplies the main robustness gain, while QA mixing adds a smaller effect after history panels are enabled.The controlled sweep identifies complementary effects from OOD supervision and structured temporal panels.
  • Memory-panel ablations: Both backbones favor 3×3 panels with one or two between-state panels; larger grids or additional panels provide no consistent improvement.The ablation supports limited between-state evidence rather than continuously increasing visual-token load, while retaining runtime panel construction as a diagnostic.
Loading 2608.15680v1…