PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation

Zixuan Chen, Lin Cao, Fenggang Zhang, Haotong Zhang, Yangtao Chen, Jieqi Shi, Jing Huo, Yang Gao

arXiv:2610.33863v1cs.RO

TL;DR

Long-horizon manipulation requires progress judgments that account for earlier interactions, since visual progress can conceal unmet task prerequisites. RoboLongR learns history-dependent rewards and uses them to guide policy adaptation; it reports improved success on held-out tasks and real-robot evaluations.

  • Problem

    Visual progress can conceal unmet task prerequisites, and identical observations may imply different outcomes depending on execution history.

    Source: §I · p. 1

  • Method

    RoboLongR combines stage-order supervision and recurrent memory to learn history-dependent rewards, then guides stage-conditioned BC-to-online-RL policy adaptation with terminal-feedback correction.

    Source: §I · pp. 1–2

  • Results

    31.8% average success on 18 disjoint long-horizon RLBench tasks exceeded the strongest evaluated baseline by 9.2 percentage points; four real-robot tasks reached 57.5% success, 32.5 percentage points above the comparison method.

    Source: §I · p. 2

  • Takeaways & Limitations

    Evaluations on held-out tasks and real robots show improved complete-task success over baselines, while reward controls probe tested order perturbations and stage-shuffle has the largest component-ablation gain.

    Source: §V · p. 8

    Synthetic shuffles can introduce boundary cues and penalize valid orders or recoveries; whole-rollout correction can suppress valid prefixes and cause reward drift.

    Source: §V · p. 8

Abstract

from arXiv · show

Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.

I. INTRODUCTION

Long-horizon tasks require progress rewards that account for prerequisites and execution history, because goal-like images can conceal skipped steps. RoboLongR combines history-sensitive reward learning with policy adaptation and evaluates the approach on held-out tasks and real robots.

I. INTRODUCTION

  • Goal-like visual states can hide missing prerequisites: closing an empty drawer may resemble task completion despite skipped opening and placement steps.Identical observations can correspond to different outcomes depending on execution history.Source: §I · p. 1
  • RoboLongR combines stage-shuffle order supervision and recurrent memory to learn history-dependent rewards, then uses them in BC-to-online-RL policy refinement with sparse-feedback correction for repeated high-scoring failures.Its introduction describes evaluation on tasks excluded from reward pretraining and on real robots, with reward controls and component ablations.Source: §I · pp. 1–2
  • 31.8% average success on 18 disjoint long-horizon RLBench tasks exceeded the strongest evaluated baseline by 9.2 percentage points; four real-robot tasks reached 57.5% success, 32.5 percentage points above the comparison method.The reward was pretrained on 88 RLBench tasks and Open-X Embodiment data.Source: §I · p. 2

II. RELATED WORK

  • Prior visual-progress rewards broaden policy-learning signals, but estimated progress alone does not establish whether earlier prerequisites were satisfied.RoboLongR addresses this distinction by combining stage-order supervision with recurrent execution memory.Source: §II · p. 2

III. METHOD

RoboLongR learns dense progress rewards from execution histories and uses them to guide policy adaptation. Its reward training combines order-sensitive supervision, recurrent memory, adaptive segmentation, and trajectory augmentation.

III. METHOD

  • RoboLongR links reward learning, demonstration stage annotation, stage-conditioned initialization, and stage-scheduled online RL, with terminal feedback identifying recurring high-scoring failures.Its dense reward maps observed prefixes and task instructions to progress scores intended to support improvement beyond demonstrations.Source: §III · p. 2, A. Problem Formulation · p. 2

A. Problem Formulation

  • Causal consistency aligns rewarded progress with prerequisites supported by observed history, pursued through stage-order supervision and terminal-feedback correction without annotated prerequisite graphs.Source: A. Problem Formulation · p. 2

B. Causally Consistent Reward Learning

  • Stage-shuffle creates order mismatches by reordering intact demonstration segments, preserving positive targets before the first displaced step and setting later targets to zero.Padding-masked MSE supervises original and sampled shuffled prefixes.Source: B. Causally Consistent Reward Learning · p. 2
  • Recurrent memory carries interaction evidence beyond the 100-frame attention window, resets between episodes, and uses only each observed prefix for prediction.The reward model uses frozen visual and instruction encoders, an autoregressive Transformer, a recurrent module, and a progress head.Source: B. Causally Consistent Reward Learning · p. 2
  • Two-round reward training first uses uniform partitions and targets, then adapts segments to predicted progress changes using per-demonstration min-max normalized predictions.Adaptive boundaries use smoothed predicted curves; if none qualifies, five uniform segments are used.Source: B. Causally Consistent Reward Learning · p. 3
  • Rewind augmentation reverses a sampled trajectory suffix while retaining source-frame targets; splice augmentation replaces a semantic stage with a compatible stage from another successful demonstration.The objective averages valid-frame errors across augmented trajectories, and normalization by eight originals preserves augmentation-sampling effects.Source: B. Causally Consistent Reward Learning · p. 3

C. Demonstration Stage Annotation

Coarse task-category stages provide shared policy-conditioning labels, distinct from the trajectory-specific adaptive segments used for reward supervision.

C. Demonstration Stage Annotation

D. Stage-Conditioned Initialization

The policy is initialized by behavior cloning and refined with PPO using history-dependent progress rewards. A stage schedule governs policy conditioning, while repeated high-scoring failures trigger reward correction.

E. Stage-Scheduled Online RL

  • PPO refines the behavior-cloned policy using the learned progress score as its per-step reward.The actor and critic condition on current observations, robot state, language, and stage labels, while reward prediction uses execution history.Source: E. Stage-Scheduled Online RL · p. 4, D. Stage-Conditioned Initialization · p. 4
  • Progress return can favor prolonged, high-scoring unfinished executions over early completion, and terminal feedback does not eliminate this incentive by construction.Episodes end at success, terminal failure, or the 500-step horizon, without a terminal bonus.Source: E. Stage-Scheduled Online RL · p. 4
  • At least 17 successes in a rolling window of 20 completed episodes advance the schedule to the next stage for the following episode; stage 1 remains active once reached.Training ends at the interaction budget without forced transitions, so runs need not visit every stage; evaluation always uses stage 1.Source: E. Stage-Scheduled Online RL · p. 4
  • Three consecutive high-scoring failures for a task trigger reward correction after the PPO update, leaving collected rewards and advantages unchanged.Correction uses queued failure triples without offline replay; terminal failure cannot localize the error, so whole-trajectory correction can suppress valid partial progress.Source: E. Stage-Scheduled Online RL · p. 4

IV. EXPERIMENTS

The experiments assess policy adaptation on tasks excluded from reward pretraining, execution-order sensitivity, component contributions, and real-robot transfer. The benchmark table reports success rates for 18 long-horizon RLBench tasks grouped by dependency strength.

IV. EXPERIMENTS

  • The evaluation asks whether learned rewards improve policies on tasks excluded from reward pretraining.The experiments also test execution-order sensitivity, component contributions, real-robot gains, and reward responses to prerequisite-violating executions.Source: §IV · pp. 4–5
  • Table I reports success rates for 18 long-horizon RLBench tasks grouped by dependency strength, with means and sample standard deviations over three seeds and 50 trials per seed.AVG. denotes the within-group mean.Source: Table I · p. 5

A. Experimental Setup

The evaluation separates reward pretraining from policy adaptation and tests complete-task success on held-out tasks. Results assess policy improvement and reward sensitivity to execution-order perturbations.

A. Experimental Setup

  • The setup uses shared demonstrations, stage-conditioned policy architecture, behavior-cloning initialization, interaction budgets, and stage schedule across RL methods.Complete-task success is evaluated independently of learned return; simulation results average three seeds with 50 randomized-layout trials per seed.Source: A. Experimental Setup · p. 5

B. Policy Improvement on RLBench

  • 31.8% mean success on 18 held-out tasks exceeds ReWiND’s 22.6% by 9.2 percentage points under the same demonstration and interaction budgets.The reward was pretrained on 88 RLBench tasks and Open-X data; each evaluation task provided 20 demonstrations for stage annotation and behavior-cloning initialization.Source: B. Policy Improvement on RLBench · p. 5, A. Experimental Setup · p. 5

C. Execution-Order Sensitivity

  • 90.0% versus 55.0% ranking accuracy and paired margins of 0.40 versus 0.04 show clearer separation of correct and order-perturbed trajectories for RoboLongR than ReWiND.Both methods distinguish rewound trajectories, while progress MSEs on correct demonstrations remain close at 0.023 and 0.021.Source: C. Execution-Order Sensitivity · p. 6

D. Ablation Studies

Ablations test how RoboLongR's components affect policy success under shared demonstrations and interaction budgets. Stage-shuffle removal produces the largest observed decrease, while schedule and segmentation controls qualify how other gains should be interpreted.

D. Ablation Studies

  • Removing stage-shuffle lowers mean success from 31.8% to 24.3%, a 7.5-percentage-point decrease and the largest observed drop.The ablations use shared demonstrations and interaction budgets.Source: Figure 4 · p. 6, D. Ablation Studies · p. 6
  • Removing online correction, recurrent memory, or the stage schedule yields 28.4%, 28.9%, and 29.8% success, respectively.These comparisons support the utility of each component within the evaluated pipeline.Source: D. Ablation Studies · p. 6
  • The no-schedule control gains 2.0 percentage points under the same interaction budget, but combines initial conditioning and subsequent scheduling.It therefore does not isolate the contribution of individual stage transitions.Source: D. Ablation Studies · p. 7
  • One, two, and three reward-training rounds yield 28.8%, 31.8%, and 32.0% success, indicating that two rounds capture most observed improvement.Coarse segments for stage-shuffle yield 29.8%, while adaptive indices used for both reward augmentation and policy conditioning yield 30.4%.Source: D. Ablation Studies · p. 7
  • The ablations favor separate stage representations: trajectory-specific partitions for reward supervision and a shared coarse vocabulary for policy conditioning.Using coarse segments for stage-shuffle reduces success to 29.8%; using adaptive indices for both functions yields 30.4%.Source: D. Ablation Studies · p. 7

E. Real-Robot Validation

On four real-robot tasks, RoboLongR outperforms ReWiND in pooled success, while held-out rollouts show clearer reward separation between valid and failed executions. The conclusion also notes limits in reward supervision and alignment.

E. Real-Robot Validation

  • The four tasks cover articulated-object interaction, tool use, multi-object manipulation, and assembly, with ten trials per task and method.A blind annotator judges complete-task success from video; failures are not selectively repeated.Source: E. Real-Robot Validation · p. 7
  • RoboLongR succeeds in 23/40 real-robot trials (57.5%) versus ReWiND's 10/40 (25.0%), a 32.5-percentage-point gain.The pooled interval describes the fixed task mixture and does not establish per-task statistical significance.Source: E. Real-Robot Validation · p. 7, Table III · p. 8
  • On held-out real-robot rollouts, both methods track valid progress, but RoboLongR shows clearer separation of valid and failed executions, especially on three tasks.ReWiND assigns substantial rewards to both valid and failed executions; raw scores are compared within each model because scales differ.Source: E. Real-Robot Validation · p. 8
  • On Board Wiping, RoboLongR's score rises modestly near the end of a failed execution but remains below its score for the valid execution.The scene resembles completed wiping despite the failed prerequisite sequence.Source: E. Real-Robot Validation · p. 8

V. CONCLUSION AND DISCUSSION

  • The conclusion reports improved complete-task success on held-out and real-robot tasks, with reward controls and component ablations supporting the evaluation.It identifies supervision and reward alignment as limitations: synthetic shuffles can penalize valid orders or recoveries, and whole-rollout correction can suppress valid prefixes and cause reward drift.Source: §V · p. 8

APPENDIX

The appendix inventories reward-pretraining tasks, specifies prerequisite relations for evaluation tasks, and describes reward-model inputs and recurrent state.

A. Reward-Pretraining Task Inventory

B. Evaluation-Task Prerequisites

  • Table V summarizes prerequisite relations and valid alternative orders for the 18 evaluation tasks, with ≺ denoting precedence.Groups are defined before policy training from RLBench success conditions and physical task structure, independently of method performance.Source: B. Evaluation-Task Prerequisites · p. 11, Table V · p. 11

A. Reward Inputs and Recurrent State

  • The reward model encodes each camera view independently, mean-pools view features before projection, and initializes its single-layer recurrent module from a zero state.Inputs are 224 × 224 center-cropped images; semantic subtask labels support augmentation and policy conditioning but are not reward-model inputs.Source: A. Reward Inputs and Recurrent State · p. 11

B. Baseline Objectives

Progress+Mono augments progress regression with stage classification and a monotonicity penalty, while the experiments fix hyperparameters using only the reward-pretraining validation split.

B. Baseline Objectives

  • ReWiND uses its released architecture and default hyperparameters within the shared policy-learning protocol.Source: B. Baseline Objectives · p. 11
  • Progress+Mono adds a stage-classification head and penalizes decreases in expected stage index between consecutive valid steps.Its stage and monotonicity terms each have unit weight; positive regression uses no shuffled negatives.Source: B. Baseline Objectives · p. 11

C. Randomization and Hyperparameter Selection

  • Hyperparameters are selected on the reward-pretraining validation split and fixed without access to evaluation-task results.Seeds control network initialization, demonstration sampling, simulator randomization, and PPO sampling.Source: C. Randomization and Hyperparameter… · p. 11

D. Real-Robot Controls and Trial Records

Real-robot trials standardize hardware and workspace conditions and record execution failures; separate diagnostics report adaptive partition statistics and note a confound in swap-distance comparisons.

D. Real-Robot Controls and Trial Records

  • The Panda uses a parallel-jaw gripper and RGB streams from two RealSense cameras, with workspace and initial-pose sampling regions fixed across methods.Source: D. Real-Robot Controls and Trial Records · p. 11
  • Controller resets, safety stops, human interventions, and hardware failures are recorded separately, with outcomes judged by complete-task success criteria.Representative failures include missed grasps, premature releases, collisions, and inaccurate placements.Source: D. Real-Robot Controls and Trial Records · p. 11

A. Adaptive Partition Statistics

  • Boundary detection is most frequent in opening and closing tasks and least frequent in composition tasks.Adaptive segment counts vary by trajectory and can exceed the uniform fallback count; Table VI reports detection frequency and mean counts.Source: A. Adaptive Partition Statistics · p. 11, Table VI · p. 11

B. Evaluation Swap Distances

  • Swap distance does not isolate execution-history sensitivity because it can also affect boundary discontinuities.The evaluated swap distances are adjacent, one-intermediate-segment, and wider separations of at least three.Source: B. Evaluation Swap Distances · p. 11

Related papers

Loading 2610.33863v1…